As fans and users of Apache Lucene/Solr, we’re excited to host the NYC Apache Lucene/Solr Meetup's roundtable this Wednesday, Feb. 12. Titled “Lucene/Solr: The Default Search Engine for Hadoop,” the meetup will feature some of the biggest Hadoop vendors in the market discussing ways to access data in a quick and predictable way that also leverages many of the features users have come to expect from search. If you’re a software engineer or other technologist who’s hoping to learn more about search and big data, this is an event for you!

LucidWorks CEO Paul Doscher will host the roundtable, which will also feature Mark Miller of Cloudera, Mike Vogel of Knowledgent, and Joseph Caserta of Caserta Concepts. More on the night’s speakers:

  • Mark Miller is a Lucene/Solr committer and PMC member as well as an Apache member. After starting with Lucene in 2006, Mark has spent most of his time getting paid to work on the open source software projects that he loves. Mark is probably best known for his work on Lucene, Solr, and SolrCloud and is currently learning all about Hadoop while focusing on search as a software engineer at Cloudera.
  • Paul Doscher: As CEO, Paul is responsible for LucidWorks’ vision and success in the enterprise-wide search, discovery and analytics market. He comes to LucidWorks with 30 years of sales, marketing and business management experience within high-tech enterprise software. 
  • Joe Caserta is president of Caserta Concepts, a New York-based technology consulting firm that builds big data solutions for businesses and organizations.
  • Mike Vogel is a senior technologist and product manager at Knowledgent Group, a consultancy that advises companies on their big data initiatives.

Hope to see you here! Food and drink provided.

Interested in Lucene/Solr, Elasticsearch and other search tools? Then come to Gilt’s Manhattan office on Monday, Nov. 18 for “Introduction to ElasticSearch,” the next event organized by the NYC Search, Discovery & Analytics meetup group. Hosted by Otis Gospodnetić­—founder of Sematext and a committer to Apache’s Lucene, Solr, Mahout, Nutch and Open Relevance projects—the meetup group is for anyone who is interested in search, discovery, analytics and related fields: information gathering, extraction, and retrieval, natural language processing, text analytics, data mining, sentiment detection, named entity recognition, and visualizations.

Monday’s featured presenter is Sematext Software Engineer Radu Gheorghe, who will discuss what Elasticsearch is and how it can act as your NoSQL data-store while providing quick, flexible and scalable search (for example, indexing logs or storing your product information so customers can search on them). Radu’s presentation includes a demo that meetup attendees can join.

There are a few spots left for the meetup, so RSVP here ASAP! Gilt will provide food and drinks.


Gilt Tech is excited to announce that on Wed 7/31, we’ll host a don’t-miss Lucene presentation featuring LucidWorks CTO/cofounder Grant Ingersoll and Gilt Senior Engineer Ugo Matrangolo. This free event will take place at TCube, 32-34 Castle St., Dublin 2 and include pizza and beer. Doors open at 6:30 PM, with presentations beginning at 7 PM. Go here to RSVP.

In addition to his role at LucidWorks, Ingersoll is a Lucene and Solr committer and long-standing member of the Apache Software Foundation. He’ll discuss the latest and greatest capabilities in Lucene and Solr related to relevance, distributed search, and faceting, and explain how to leverage these capabilities to build fast, efficient, scalable, next-generation data driven applications. Matrangolo, a member of Gilt’s Stores Team, has played a vital role in developing and enhancing the Gilt’s search capabilities, and will focus on how Solr has enabled Gilt to provide a controlled and targeted search experience for its millions of customers in the face of intense, daily spikes in site traffic.

Please join us for what promises to be an enlightening and lively discussion!


What is your role at Gilt?

I’m lead engineer for Team Galactus. (We like to give our teams abstract names that don’t refer to a particular initiative; at the time of forming Galactus, we were going through a comic book phase!) I’m a full-stack engineer, but the vast majority of my work at Gilt is on the backend: developing shared libraries, web applications and services.

How long have you been with Gilt?

Just over two years.

Has your role here changed over time?

No, I’ve always been a backend engineer. I’ve done some frontend stuff with previous companies that were very small, and where you had to roll up your sleeves and basically do everything.

Why do you enjoy backend development more than frontend?

It’s less like the Wild West than frontend development. You can implement clean, well-tested and reusable solutions fairly easily; it’s a lot more difficult to do that in the frontend. Frontend work often feels like one hack on top of another hack, with a third hack thrown in to work around some IE issue. And it all feels really fragile, because it’s not statically typed. Refactoring something can be a very stressful experience! There have been a few frameworks over the years built to try to solve this problem (e.g. GWT), but it’s still generally a more challenging environment to work in. Throw CSS into the mix, which can do some really cool, untestable stuff, and … well, you get the picture.

Which technologies do you use on a day-to-day basis?

Scala, Play, Handlebars (both the standard JS Handlebars and Gilt’s own Scala implementation—fondly called Scandlebars, and written by Gilt engineer Mark Wunsch.

What’s your personal strategy for learning a new technology?

My inclination is always to just learn on the job. That only gets you so far, though. And when you’re working on important pieces of software, you need to be the expert in the technologies you are using. Taking time out to really learn those technologies is something I have to work hard at; it’s not something that comes naturally. My preference when I do need to learn something in detail is a good old-fashioned paper book—remember those?

I recently took Martin Odersky’s Coursera course on Scala. This was really enjoyable.

Describe for us some of the projects that you’ve been working on recently.

The team is currently in the process of migrating our sale listing page (the most visited page on Gilt: ~30k rpm at noon) over to a new technology stack: Play, Scala, Handlebars, and Solr. As part of that work we are building a new app and adding a lot of new code to shared libraries.

We are also working to integrate Scalatest/Selenium testing into our continuous delivery framework. Continuous delivery allows us to deploy to production in minutes at any time with no manual interaction after the developer types “sbt release” on their laptop. Having really good Selenium test coverage, written using Scalatest, is a key part of that.

A small side project I’ve just completed is a piece of code that allows you to build “views” of a cache. You simply provide a transformation function, and you get back a virtual cache of a different type. The views themselves can be cached (if the transformation is expensive) and you can stack them, so you can have views of views. Getting the type system to work was a bit of a mind melt, but I think the end result is pretty cool.

One technology you’ve been using lately is Solr. Tell us more about that.

We started using Solr to power some basic search capabilities on the site in 2011. Galactus then built keyword search using the same technology stack. We are now moving the sale listing page onto Solr—after all, a sale listing is simply a search for all products with a particular sale ID. This means that less code is required to power the site. As part of moving the listing page over, we’re really having to step up the performance of the search technology stack. It has to be scalable and really fast!

What’s been your biggest accomplishment here?

I’ve done a few things that I’m proud of. Early on, I think my biggest accomplishment was “getting” the Gilt technology stack so quickly, and being able to contribute within a few weeks of joining. I’ve since worked on some pretty core pieces of software like Commons, which is a client library for Gilt’s domain model; the sale targeting engine, which targets sales to the appropriate members; and keyword search. I also recently led the redevelopment of the product page: a complete rewrite of the old page from the bottom up.

What would you say are the best parts about working at Gilt? And more specifically, about working at Gilt-Dublin?

The pace—things move very fast all the time. We’re on the cutting edge in terms of continuous delivery, the technologies we’re using, and some of the testing tools we’re building.

Also, the people. Gilt has some great people in so many different areas. I’ve learnt more in my first two years at Gilt than I did in the previous eight years since I left college. The Dublin office, being a bit younger than Gilt NYC, feels a little more like a start-up in some ways. There’s a really good atmosphere. Despite being the new kid on the block, we’re solving big problems and we’re doing great work!

The actual office itself is also fantastic: A penthouse office in the city centre with great views. Plus it’s in Dublin, the best city in the world!

Can you talk a bit about mentoring and your on-the-job inspiration sources?

While Gilt is using some really cool and interesting technologies, I’m an engineer rather than a technologist, so solving problems is what makes me want to go to work each day—and Gilt has some really hard, interesting problems to solve.

I’m also lucky to work with exceptional people. This is also a great inspiration: working with people who, in pretty much every interaction I have with them, teach me something.

Some of our users may have noticed a new and welcome addition to Gilt over the last few weeks—we’ve rolled out search across our Women’s, Men’s, Kids and Home stores! In Gilt Tech, we know how much our members love the thrill of the hunt when our sales go live every day; and, as a tech team, we’ve always felt good about building our site to drive a great flash-sale shopping experience. However, we also felt strongly about providing a way for letting our members find the gorgeous stuff they love faster. With this in mind, we’ve built out our keyword search using Scala, Play Framework and Apache Solr / Lucene, and have learnt a huge amount of stuff along the way.

First: a tour. If you’re one of the lucky few in our A/B test group, you’ll see the ‘magnifying glass’ search control on the top right of your screen, just below your cart / checkout:

Find the search control beneath the cart/checkout buttons

Click on the magnifying glass, and you’ll see our search box playfully swish out; you can select a store to search and then type away happily. We autosuggest on our favorite brands, colors and all of our live sale products. You can then filter by category, brand, size and color using the facet controls on the left to find exactly what you want.

Our new search results page with autocomplete

Implementing search for a flash-sales business, where inventory is limited and changes rapidly as our sales go live to a stampede of members every noon, poses a number of technical challenges: with small numbers of fast-moving inventory, we’re keen to make sure that the search results you see are genuinely the products that we have to sell at the moment you search. To make it work, we index our live products using Apache Solr, and sort and filter search results with real-time inventory status using a set of custom Solr extensions.

The search listing page is implemented in Scala using Play Framework. We’re loving Scala at Gilt, and while our growing corpus of Scala code plays very nicely with our existing Java infrastructure, it made sense to take a look at Play Framework as a new platform for Scala-based web-apps. Regular readers of the Gilt Tech blog will have seen how the Gilt Live team recently released an awesome social shopping experience using Play Framework and Web Sockets. For Search, we use Play’s templating language to render our initial listings from the client-side, and then use a combination of Backbone.js and Handlebars to handle the subsequent AJAX rendering within the browser on the client-side. The result is super fast! We found the Play stack fun and easy to use; so much so that we had time to implement some really neat features: for example, when you hover over the Next or Previous page links on a listing, we preemptively initiate a request so that the data is already there for you if you decide to click.

We feel really good about the solution, and we’re seeing our members respond really well to the feature! Happily, we have lots of great ideas and are going to continue to iterate on the experience.

Team Galactus group photo

We hope you like it as much as we do!

Adrian Trenaman, Mark Wunsch
Team Galactus, Gilt Tech