Gilt Technologie

Friday Fun with Gilt Games

ckolderup — Fri, 04 May 2012 09:17:13 -0400

The Gilt Public API is a great tool for developing new ways for people to shop quickly, efficiently, or on new platforms. But developers using the API are also finding unexpected things to do with Gilt data— they’re using it to make games.

So in the spirit of Friday, why not spend a few minutes playing with a couple of games that have been built on top of the Public API?

Gilt Memory was built by Karl Norling, a developer here at Gilt. See how quickly you can find the matching pairs of products, then challenge your friends to beat your score!

The Price is Right on Gilt was built overnight at the NYC Powered by MongoDB Hackathon last weekend. This live multiplayer game lets you and up to 3 other friends play a price-guessing game to see who knows Gilt’s products the best. Developer Yufei Liu built the project using Node.js and MongoDB.

Happy Friday, everyone!

The 5th Floor

giltquinn — Sat, 28 Apr 2012 16:23:00 -0400

Gilt Tech have been enjoying their new space on the 5th floor of 2 Park Avenue, NYC.

Gilt Tech @ NYC Powered By MongoDB Hackathon

ckolderup — Fri, 20 Apr 2012 11:45:00 -0400

What’s better than spending 24 hours in Manhattan hacking on sweet projects using MongoDB and the Gilt API? Doing it while supporting HackNY, a New York City organization that promotes innovation and entrepreneurship among the city’s up-and-coming students and hackers. And that’s exactly what you can do next weekend, April 27-28, at 10gen’s MongoDB Hackathon. A lot of great people, and great companies, are going to be there, and we’re excited to see you come down and make the next cool project. On Saturday night an esteemed panel of judges will choose winning projects and award all manner of prizes, including iPads, Xbox 360s, and more.

We’ll be at the event to share how to get started with the Public API and answer any questions you may have, give you ideas on what you could build, and maybe give you a sneak peek into what we’re working on for the future of the API, so come find us!

For questions about the event, you can send a tweet to @MongoDB. For questions about the API, get in touch @GiltTech or send an email to api@gilt.com!

The PMO: Right Work, Right Time, Right People

giltheather — Mon, 16 Apr 2012 12:16:45 -0400

You might be surprised to learn that Gilt Tech has a Program Management Organization (aka “The PMO”).

Why do I think you might be surprised? Well, the role of project management in traditional organizations has typically been one associated with paperwork and process (We need to talk about your TPS reports…). It has also been associated with hindering rapid development and slowing things down. These negative connotations are behind the reasons why many people struggle with the idea of having a PMO or a project manager on a team.

We strive to change this. We believe every organization and team should invest in project management if you want to work smarter and spend more time on what really matters. Here’s some reasons why…

One of the many tenets we are taught in our training is the “Triangle of Truth”. The project manager’s key function is historically described as making sure a project meets its objectives, is on time and on budget.
What happens more often than not is that a project manager is so focused on driving to deadline, they lose sight of the big picture. I have often heard project managers and others describe their job as “my responsibility is to make sure we hit our schedule no matter what” — and this is dangerous thinking. Our function is so much bigger than maintaining a list of tickets on schedule.

The project manager’s responsibility is to make sure we are doing the right work at the right time and with the right people. Let’s break this down a bit.

Right Work
A good project manager doesn’t simply take orders and execute on them. We have the authority to ensure we are working on what will bring the biggest contribution to the organization, and constantly question and re-evaluate this based on a combination of hard data, an understanding of the business we are working in, and good intuition.

Is this the most important thing we could be doing right now? If it isn’t, how can we bring the right people together to negotiate and agree to the right work?
How will this work both positively and negatively impact others?
Is the work defined enough that we feel confident in its execution? What is the Minimum Viable Product? Is the work actionable?
What does the data tell us and what decisions need to be made based on this data?

Right Time
A good project manager needs to have an understanding of both internal and external factors behind the schedule, and make decisions based on a deeper understanding of what is driving the timeline, maximize efficiency, and take the initiative to make adjustments along the way.

What are the market conditions like? What are our competitors doing? Is it the right time to be launching this kind of product?
Are we managing time appropriately? Is the team focused and if not, what can we do to eliminate the noise?
Are we working in the most efficient way? What changes to our current process need to be made to maximize our efficiency?
Are there deadlines that are fixed due to events such as press releases, end of quarter, or financial reporting needs?

Right People
A good project manager is also a people manager. Yes, I’m adjusting the “Cost” part of the triangle to represent people — because people’s time = money. A good project manager knows that a primary function is to alleviate any roadblocks that get in the way of your team focusing on the most important work, even if this means picking up the slack yourself. We are also responsible for team building, encouraging growth and learning, and are often the public face or “go-to person” on the team.

Do you have the right people on the team to accomplish your mission? If not, what can you do to get the people you need?
Are the right communication methods in place so that all of the people involved in the project have a clear understanding of what is happening? Do all of the people outside of the project have access to information?
Do all these people really need to be at this meeting? What can we do to eliminate unnecessary meetings for people so they can focus?
Is the team happy, engaged, and enjoying their work? What can we do or change to maximize happiness and productivity?

As you can see, there’s a lot more to project management than managing the schedule, taking notes and executing orders. Project managers oversee the end-to-end execution of a project, making sure we are doing the right work at the right time and with the right people. We are experts in efficiency, negotiation, communication, strategy and big picture thinking with the ability to break the big things down into actionable chunks. We not only get things done — we get the right things done in the best way possible.

Lessons in Ratatouille

giltheather — Wed, 14 Mar 2012 09:37:36 -0400

I have a son who is obsessed with Pixar movies, so I have seen “Ratatouille” about 100 times. It wasn’t until recently however, that I started to draw parallels between the movie’s lessons and lessons in organizational culture.

Lesson One: “Anyone Can Cook”
The core vision statement of Gusteau, the original chef who built his restaurant into an empire and what inspired a rat to pursue his dream to become a cook. There is debate around this vision within the movie — just because anyone can cook it doesn’t mean they should. Some people are better at this than others, and not every creation is going to make the menu. The point is that the vision is simple, aspirational, and inspiring.

Lesson Two: A gift is not enough — you need a passion. And mentors.
Remy the rat has a gift for being able to smell whatever ingredients are in something. He could have easily resigned to the comfortable position of poison checker for his rat clan. But he was soon bored — he wanted to create something. To add to this world instead of taking from it. You have people in our organization with gifts. You have people with passion. You need to mentor these people (as imaginary Gusteau and Colette did for Remy) so they are able to express their passion creatively.

Lesson Three: Complacency = “Follow the Recipe”
With Gusteau dead, the new leaders at the restaurant have instructed the team to simply “follow the recipe”. Gusteau was paid to create — they are paid to follow what Gusteau already did and not improvise. Seems pretty logical — just keep doing what you know works and you continue on the same trajectory. Except the customers are getting bored. They want to know what does the chef have that is new. Organizations are in danger of falling into “follow the recipe” complacency where talented people are not willing to rock the boat due to fear of something or because they feel defeated, or simply restricted from making their own improvisations.

Lesson Four: Frozen Foods might make you a lot of money, but at what cost to morale?
The new chef in charge is highly focused on making more money in the fastest, easiest way possible. In the movie, it is by expanding Gusteau’s recipes into a pre-packaged, frozen-foods line. Will it bring in revenue? Yes. Is anyone inspired? No. The team feels like what they are doing is sacrificing the quality of their work, and killing the passion in their hearts that drives them to create every day. In fact, there is a point in the movie where this plan is disbanded and the team rallies around a bonfire of the frozen foods and marketing materials and toasts to a new era of getting back to making real food for real people.

Lesson Five: “You are the boss. Inspire them.”
In the movie the team is about to face their toughest food critic who can make or break their restaurant. Linguini asks Colette “What should I do?”. She replies, “You are the boss. Inspire them.” I’m sure you all have teams of people who report to you, who look to you for guidance, mentorship, and they take their cues from what you do and what you say. It is your responsibility to inspire them, to ignite the passion that made them want to work there to begin with. If you don’t believe it — if you don’t feel it yourself you can’t expect them to believe and feel it too.

Lesson Six: Give the customers something new by iterating on what already is.
The best dishes were iterations on existing recipes, not completely new ideas. How can you improve and innovate in simple ways to get some quick wins with measurable impact? What can you do in the short term that is actionable and achievable but still innovative?

Lesson Seven: Take off the toke and give your tiny chef the credit.
Remy grows continuously frustrated because he never receives recognition or credit for his work. (He spends most of the movie hiding under the toke of Linguini, telling him what to do.) Every company needs to do more in this area. A lot of work goes on behind the scenes that many people don’t see (and then they start to question the value of it). Bring more of this to the forefront. Thank people more, publicly recognize them, call them out for even the small things. Foster a culture of “I have your back.”. Remy is a rat — you can imagine what happened when people found out he was the cook behind all the innovation in the kitchen. Recognize people for taking initiative and have their back regardless if the outcome is good or bad.

Lesson Eight: Connect with people on an emotional level.
This goes for both your employees and the customers you have. At the end of the movie there is this wonderful scene when the big bad critic guy tastes the dish of ratatouille prepared for him and he flashes back to an emotional moment where he is a boy being comforted by his mother after having gotten hurt falling off his bike. The food connected with him emotionally. Your products can do this too. You can do this with your staff. Listen, empathize, connect, guide, inspire.

If anything, I hope you go watch the movie. :-)

SummerQAmp: introducing QA as a career opportunity

klhaggard — Wed, 07 Mar 2012 16:21:04 -0500

The tech team at Gilt is excited to launch SummerQAmp (summer-kamp) with Bon Jovi, the White House, GroupMe and Onswipe. This is a great initiative that is launching as a part of the White House’s Summer Jobs+ program to introduce Quality Assurance as a career path to youth in the U.S.

QA @ Gilt

Our Quality Assurance team is made up of individuals that are programmers with Computer Science degrees to others that started out in our Customer Support team without a technology degree.

We love to automate tests but we also understand the value of manual testing. One thing that is common among the team members is that they are passionate about what they do and know the products they are testing better than anyone. We also work with a great offshore team and they are led by our QA team in the U.S. We see the value of outsourcing as a way to extend our business hours, not to replace jobs in the U.S. There is nothing better than having an engineers and testers working side-by-side.

Our SummerQAmp Goal

By helping launch this program and participating in it, we hope to teach the values of Quality Assurance and demonstrate that it can be a career or a gateway to other areas within technology. Although, we really hope their passion for breaking things will keep them wanting to test.

We also hope with this program that we can give people that may not know how to get started or the have financial means to get started in technology a chance to gain exposure to the skills necessary to be successful in a technology career.

Get involved by signing your company up at http://www.summerqamp.org.

Public API Language Bindings, Round 2

ckolderup — Thu, 01 Mar 2012 15:04:27 -0500

There’s been even more work done getting the Gilt API wrapped up in language bindings over the past two weeks and I’m here to present the newest to you in case any of them tickle your fancy.

gilt_api_php is a PHP client written by Gilt Tech member Dan Revel. This library has tests and a demo app to get you started on downloading and featuring sales and products in no time.

Gilt-Java, by Gilt Tech’s very own Nitin Dhar, is of course a Java library for the Public API. Nitin is working on some Android compatibility so mobile app developers don’t need to remain glued to our iOS bindings!

gilt-python is a brand new Python library brewed here at Gilt by engineer Andrew Ellerton. If you’re working in Python, give it a try and see how you like it!

We’re hard at work on compiling a permanent directory of client libraries like the ones we’ve featured on this blog and some of the applications people have built already. Email us soon if you have something you want to see included!

Affiliate Marketing Program Open to Developers

ckolderup — Thu, 23 Feb 2012 13:11:16 -0500

It’s been great to hear from developers over the past two weeks about their plans for the Gilt Public API, and we can only imagine what’s in the works out there that we haven’t heard about yet! Meanwhile, back at Gilt HQ, we’re hard at work on adding more to the API— more data and more ways of accessing what you want in the easiest way possible.

In the meantime, though, I’m happy to announce that we’ve finally got everything in place to enable you to participate in the affiliate marketing program. By enrolling in this program and adding a single parameter to all the requests you pass in, you can earn a competitive 6% commission on all sales from referred customers. When you consider that the luxury items on Gilt lead to average total order values of $150, that adds up pretty quickly.

All the information you need to get started has been added to the Gilt Developer Portal; the technical information is on the main documentation page and information about how to sign up for the affiliate program is available on our FAQ. What are you waiting for?

First Public API Language Bindings Available Now!

ckolderup — Wed, 15 Feb 2012 11:52:00 -0500

7 days ago, we launched our Public API and the accompanying Developer Portal. We’ve received a lot of great feedback, updated the Portal a bit to better give you the information you need, and talked to a lot of developers with some really neat ideas for ways to use the API. Now’s a great time to get started on your own application!

Today I’m here to share some of the hard work that people have been doing, both here at Gilt and in the developer community at large, to make the API more accessible to developers of specific languages. We now have the first API language bindings available for Javascript, Ruby, Objective-C, Scala, and Python!

There’s more to come; I’ll make sure to update here when we have support for more languages or other libraries that might tickle your fancy. Until then, this should be enough to get you started, right?

Javascript

Developed by Chris Young-Zawada here at Gilt, these Javascript language bindings are available as a jQuery-based implementation, so all you have to do is drop in a recent version of jQuery and this .js file and you’re off to the races!

Ruby

gem install kin is all you need to get started with using the Gilt API in Ruby. Developer Vivek Bhagwat put together this rubygem to let Ruby developers worldwide have access to Gilt’s great flash sales in their applications. Thanks, Vivek!

Objective-C

iOS development is kind of a big deal right now, and what better way to dip your toes in the water or extend the reach of your mobile app empire than to use this Gilt API iOS client library to integrate Gilt sales data into an app for iPhone, iPod Touch and/or iPad? (That’s a rhetorical question. This is the ultimate way to do those things.) The SDK was put together by a crack team of Gilt mobile developers: Louis Vera and Adam Kaplan.

Scala

Recently among my friends and, it seems, in the developer community at large, what started as a whisper (“Scala is pretty sweet”) has turned into a roar (“DEPLOY ALL THE SCALA”). If you’re one of the converts, I’m happy to tell you that developers Moses Nakamura and Vivek Bhagwat have prepared a fully-tested, simple to use library called Aurum that give you access to the Gilt API. Moses and Vivek win the prize for being the first client library developed outside of Gilt— it was up and ready for use fewer than 48 hours after the launch of the API!

Python

Not content to rest after that, Moses put together a simple Python wrapper for the Gilt API as well. It’s currently the slimmest of the client libraries highlighted here today, clocking in at just over 100 lines of code, but supports digging through all of the sales and product information that you need to explore our curated sales and do all kinds of fun stuff with them. Good luck, Pythonistas!

Managers! Become the Flywheel.

giltquinn — Mon, 13 Feb 2012 13:26:00 -0500

Arguably, the job of an engineering manager is to hire and sculpt a development team that is not only highly productive, but also precisely resourced for immediate business priorities.

Easier said than done. Businesses are highly erratic organisms. Especially start-ups. From week to week our business environment changes and company priorities evolve accordingly. In response to this, so does the position of senior management on the best way to adapt.

Senior managers flourish in these rough seas, happily charting new courses as new storms brew.

Development teams don’t fare as well when deprived of continuity. Reorganizing teams and re-prioritizing efforts at the first sign of trouble is problematic for a few reasons:

Engineers can’t be thought of as fungible resources that can easily skip between teams. Optimal team dynamics form slowly. Interactions take a while to mature and processes need time to be adapted to the specific needs of the team.

Great products don’t happen overnight either. They typically result from several iterations by stable teams. The first incarnation of any effort is almost always sub-par. This is as it should be. As Reid Hoffman said: “If you’re not embarrassed by your first product release, you’ve released too late”. The real magic happens after the first or second iteration when the team really starts understanding what the problem is and has acquired the domain knowledge and built the tools to attack it.

So how should you get the most of out your team in an unstable environment?

Balance the team: When the ratio between engineering team size to the rest of the company is too small, you create the capacity to generate many more ideas than can possibly be implemented, or even discussed fully. This results in endless distractions and prioritization sessions, and reduces critical focus on the work currently being done.

There are a few models for creating balance. One is to simply drive everything from the tech team, for example GitHub has zero managers and developers are 100% responsible for the product direction. This isn’t an option for most companies, but creating a engineering team that is appropriately resourced for the overall organization is.

Maintain Focus: Create a North Star for engineering teams by creating, communicating and repeating a strategy for your company and product. Define KPIs to track progress against the strategy. Stay focused on these KPIs for at least a quarter before iterating the strategy.

Limit Distractions: A colleague recently reminded me of a great quote from The Mythical Man Month: “the best engineering manager I ever saw served often as a giant flywheel, his inertia damping the fluctuations that came from the market and management people”.

Be the flywheel and keep your team focused on your strategy. Limit non-essential or off-track meetings and help cross functional teams focus and work tightly on active initiatives. Project rooms allowing face-to-face communication and tight teamwork can be extremely effective.

Empower and Encourage Ownership: Encourage teams to take complete, long-term ownership of initiatives and the artifacts (code) that they produce. Adopt tools that help reinforce ownership while maintaining flexibility (we use Gerrit at Gilt). Defend the team from the inevitable onslaught of requests to “re-prioritize” their current work that reduce ownership and focus.

Doing this isn’t easy, but as a managers our responsibility, perhaps our biggest responsibility is to make sure we have a strategy, to repeat it until people are tired hearing it and to do everything we can to maintain appropriate inertia around it.

Quinn.

Follow Gilt’s awesome tech team on twitter @gilttech and read about our new API at http://dev.gilt.com

NoSQL in the Real World - The Video, Pics and Slides

giltquinn — Fri, 10 Feb 2012 13:52:00 -0500

Thanks to everyone who came to Gilt Tech’s latest tech talk, NoSQL in the Real World.

We had a great series of speakers including:

Ara Anjargolian - Redis
Matt Parker - CouchDB
Sean Cribbs - Riak
Edward Capriolo - Cassandra
Luke Gotszling - MongoDB

Huge thanks to Rockman and Maureen for organizing the event, and to AOL ventures for sponsoring and hosting.

Enjoy out the video …

The Pictures …

The Slides …

Riak

Redis

Cassandra

Membase and MongoDB

CouchDB

Gilt Public API Launched

ckolderup — Wed, 08 Feb 2012 10:13:24 -0500

I’m really excited to tell you that we’ve launched our Public API today, providing programmatic access to a constantly-updating collection of flash sales full of the usual kind of awesome stuff we offer on Gilt’s stores.

We held an internal hackathon two weeks ago and saw some really interesting projects come out of just a day’s work and the brainpower of one room of (admittedly super-smart and obviously very handsome) Gilt engineers, but I can only imagine where things can go now that we’ve got the whole developer community armed and ready.

If you’re not like me and don’t just get pumped up whenever you see a new API you can curl, maybe I can sweeten the deal a bit— we’re in the process of putting the finishing touches on an affiliate marketing program, meaning that you can earn a cut of any money we make when your app leads to a purchase on Gilt. You can start building your project now and we’ll have the details on how to sign up and integrate this into your work soon.

The Gilt Developer Portal is your go-to site to read up on how to use our API and apply for API keys for your applications. We’ll be adding more information to that site regularly as we add new features and content to the Public API. Exposing sales and products is just a first step; we’ve got a lot of exciting ideas in the works, so stay tuned!

Old Browsers in eCommerce: What to do with IE6?

etotheipi1 — Tue, 07 Feb 2012 21:34:00 -0500

In a humanitarian effort to improve developer quality of life, we at Gilt recently stopped supporting Internet Explorer 6. This means we no longer use, develop or test for the browser. Given the small and shrinking population of IE6 users and the auto-upgrades planned for this year, we felt that it is worth our developers’ happiness (and sanity) to abandon support efforts. Our FEET (Front End Engineering Team) are certainly happy to let go of ‘[if IE 6]’ conditional comment hackery and other ugly shoehorning techniques.

We found ourselves wondering… Would IE6 users have a better shopping experience if we redirected them to the minimalistic mobile version of gilt.com? Could the simple no frills layout designed for mobile devices just might be an improvement over the fully-featured yet unsupported site for IE6 users? Let’s walk through the exercise of answering this question, from data collection to results summarization.

Split Testing

Most often the best way to answer a question is with an experiment. In the context of Gilt and e-commerce, we find split testing to be an effective method for assessing the effect of changes (treatments) to the user shopping experience, be it the reorganization of a sale page or a modification to our checkout flow. Controlled split tests allow us to gauge the impact on our KPIs with statistical confidence, relying less on opinion and more on evidence-based reasoning.

The idea is to separate a representative subset of the user-base to expose the change to and assess the effects of the change on this smaller group before making the decision to roll out to all visitors. This ‘test’ subset should be paired with a mutually exclusive ‘control’ subset. Both subsets should be randomly sampled such that they are characteristically the same as each other. We may or may not choose to limit the testing to a segment of our users. For instance, we may want to limit the test to NY members, in which case our ‘test’ and ‘control’ groups should only be chosen from members in the NY population.

Comparing Conversion Rates

If the redirect has any notable effect on the user experience, it should show in the proportion of buyers-to-visitors, a.k.a. ‘visitor conversion rate’. Conversion rates are proportions and from a statistical point of view are quite easy to work with. We will be comparing the conversion rate of a ‘Test’ group that sees the mobile site to a ‘Control’ group that sees the regular site. For sake of brevity, I will not go into detail about the nuances of measuring our numerous other site metrics (revenue, dwell time, drop-off… etc.) and test scenarios (multi-variate, overlapping, longitudinal… etc.).

On a high-level, we want to accomplish the following steps:

Segment out our IE6 visitors - our ‘IE6 base’.
Randomly sample a representative test group from the IE6 base and redirect them to m.gilt.com.
Measure the difference in ‘visitor conversion’ between the test group and the remainder of the IE6 base (control).

Test or control group membership is tracked via a randomly assigned ‘partition’ number (integer) that is issued on first visit in a cookie and persisted thereafter in our operational database. We can then redirect IE6 users to m.gilt.com based on the partition they have and whether the user-agent string indicates they are running IE6. The mobile version of our site is very lean and lacks javascript tagging for tracking site interactions. As a result, we will need to fall back to some good old ad-hoc data munging from our server logs and match up against our OLTP database to piece together the conversion rates per group.

We can segment out IE6 visits quite easily by checking the GET request user-agent string in our server logs. The string should contain ‘MSIE 6.0’.

Example user-agent string: ‘Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)’.

Splunk is a great tool for efficiently parsing and aggregating statistics from log files. We can use it find unique IE6 visitors by day (our IE6 base group) with the following query:

Splunk query 1 (output filename: all_ie6_users.csv):

host="olb*" guid "compatible; MSIE 6.0" | stats count by guid, date_mday, date_month

Half of these users belong to our test group and were randomly sectioned out and redirected to mobile. For this test we added logging to allow us to find those visitors from the test group that were redirected with the search string ‘IE6 mobile redirect’.

Splunk query 2 (output filename: test_ie6_users.csv):

host="olb*" guid "IE6 mobile redirect" | stats count by guid, date_mday, date_month

Note: “olb*” limits the search to our public facing load balancers that are taking care of the redirection and ‘guid’ is a unique identifier for each user.

The flat-files returned by Splunk are structured like so:

 guid             | month_day |  month   |  count  
------------------+-----------+----------+---------
 00e020bc-ccc8... |         3 | december |      35
 00ea4710-5be5... |         2 | december |      12
 0123d860-34aa... |         1 | december |       8

The results of these queries tell us who belongs in our test and control groups and when they visited. Users from Query 2 are ‘Test’ and users from Query 1 that were not also in Query 2 are ‘Control’. At this point I will move these files into temporary tables on a copy of our Postgres transactional database.

create temp table all_ie6_users (guid varchar(500), month_day integer, month varchar(50), count integer); 
\copy all_ie6_users from all_ie6_users.csv with delimiter ',' csv header 
create temp table test_ie6_users (guid varchar(500), month_day integer, month varchar(50), count integer);
\copy test_ie6_users from test_ie6_users.csv with delimiter ',' csv header

These two tables can be combined to produce one recordset, with a label for ‘Test’ or ‘Control’ and a standard date timestamp.

create temp table labeled_ie6_users as
select
  date(all_ie6_users.month || ' ' || all_ie6_users.month_day || ', 2011') as day_timestamp,
  (case when test_ie6_users.guid is null then 'Control' else 'Test' end) as test_group,
  all_ie6_users.guid
from all_ie6_users left join test_ie6_users
  on all_ie6_users.guid = test_ie6_users.guid
  and all_ie6_users.month_day = test_ie6_users.month_day
  and all_ie6_users.month = test_ie6_users.month;

select * from labeled_ie6_users limit 3;
 test_group |    day_timestamp    | guid             
------------+---------------------+------------------
 Test       | 2011-12-01 00:00:00 | 00e020bc-ccc8... 
 Control    | 2011-12-02 00:00:00 | 00ea4710-5be5... 
 Control    | 2011-12-02 00:00:00 | 0123d860-34aa...

Our ingredients in place, we can now calculate conversion rates per group by cross-referencing the guids and dates with our transaction records and counting unique buyers and visitors per day. These daily counts are then aggregated into a final ‘Daily Visitor Conversion Rate’ per test group. Note, a visitor/buyer is only counted once per day even if they visit/purchase multiple times, a bit of precision lost because we lack mobile site page view data.

 test_group | buyers | visitors | conversion_rate
------------+--------+----------+-------------------
 Test       | 1154   | 36794    | .03136381
 Control    | 1749   | 47275    | .0369963

Basis Point Difference: (.03136381 - .0369963) = -0.00563249
% Lift, Test over Control: -0.00563249/.0369963 = -0.1522447 = -15.22%

The ‘% Lift’ value is what we really care about. It tells us that 15.22% fewer daily visitors from the test group made a purchase relative to the control group.

Do we trust that this difference has anything to do with the redirect treatment? How do we know that it is not just random noise?

We can calculate a ‘95% confidence interval’ to quantify a range where the ‘real’ difference in conversion is likely to be. A confidence interval provides us with a lower and upper bound on where we would expect the difference in conversion rate to fall if we continued the test indefinitely. The more evidence (data) we have, the narrower the confidence interval. The width of the interval is a function of the size of our sample (# visitors) and the magnitude of the absolute difference between our test and control proportions (Lift %). With a larger sample and/or absolute difference, we get a tighter range. For example, a 10% difference between two samples of 10M observations will have a much narrower interval than a 1% difference between two samples of 100 observations.

Calculating a 95% Confidence Interval

R is an open source statistical computing language (technically a dialect of S-Plus) with a robust collection of libraries. Below is some R code for producing confidence limits around the difference of two proportions. We make use of the handy prop.test() (Proportions Test) function for the heavy lifting in our calculation.

test_numerator <- 1154
test_denominator <- 36794
ctrl_numerator <- 1749
ctrl_denominator <- 47275
test_rate = test_numerator/test_denominator
ctrl_rate = ctrl_numerator/ctrl_denominator


model <- prop.test(x=c(as.numeric(test_numerator),as.numeric(ctrl_numerator)), n=c(as.numeric(test_denominator),as.numeric(ctrl_denominator)), conf.level = .95, correct=F)

model_coef <- (model$estimate[1] - model$estimate[2]) / model$estimate[2]

prop_results <-
  data.frame(
   test_value = test_rate,
   ctrl_value = ctrl_rate,
   test_lift = model_coef,
   conf_int_low = model$conf.int[1]/model$estimate[2],
   conf_int_hi = model$conf.int[2]/model$estimate[2],
   row.names = NULL
 )

> prop_results
  test_value ctrl_value  test_lift conf_int_low conf_int_hi
1 0.03136381  0.0369963 -0.1522447   -0.2194746 -0.08501468

Final Results:


     Test Visitor Conversion Rate:  3.136% (1154/36794)
  Control Visitor Conversion Rate:  3.7% (1749/47275)
         Test Lift over Control %:  -15.224 %
Test Lift 95% Confidence Interval:  [-21.947% to -8.501%]

Based on the results, we can be fairly certain that IE6 users redirected to our mobile site buy less often. 8.5% - 22% fewer visitors make a purchase if pointed to the mobile site. In stats lingo, our result is ‘significant’ at 95% confidence because 0 is not between -.22 and -.085. In other words, we are 95% confident that the difference between test and control is not positive. If you remember p-values from statistics 101, a 95% confidence interval that does not include 0 is equivalent to a p-value < 0.05. Not only can we say the effect is significantly negative but we can also speak to just how negative the effect is, which is lacking in the typical NHST approach to testing which gives you a yes/no indication of difference rather than a magnitude.

Keep in mind, there’s no free lunch and intepretting results can be tricky. A counterintuitive result may leave you with more unanswered questions than you started with and ‘significant’ does not necessarily mean ‘important’ if the magnitude is trivial. Nonetheless, doing the work of interpreting the data is a worthwhile learning experience and a robust split testing methodology is an invaluable tool for decision making. When working in teams, everyone has an opinion but as the statistician W. Edwards Deming is often quoted, “In God we trust; all others must bring data.”

Read on for more detail on how the proportions test works with sample code…

Now you may be wondering what exactly is going on behind the scenes for the prop.test() function to produce our confidence interval. The calculation is fairly straightforward and can be implemented in virtually any language capable of simple math.

Components:

Z-score = 1.96 for 95% confidence (if significance is being calculated repeatedly, you will want to correct for multiple comparisons with something like a Bonferroni correction)
Test Numerator = #buyers from the test group
Test Denominator = #visitors from the test group
Control Numerator = #buyers from the control group
Control Denominator = #visitors from the control group

Here is a decomposition of what the prop.test() is doing in R, borrowed from the source code and simplified for our use case:

  test_numerator <- 1154
  test_denominator <- 36794
  ctrl_numerator <- 1749
  ctrl_denominator <- 47275
  test_rate = test_numerator/test_denominator
  ctrl_rate = ctrl_numerator/ctrl_denominator

  x=c(as.numeric(test_numerator),as.numeric(ctrl_numerator))
  n=c(as.numeric(test_denominator),as.numeric(ctrl_denominator))
  conf.level = .95
  zscore = qnorm((1 + conf.level)/2)

  estimate <- x/n

  DELTA <- estimate[1L] - estimate[2L]
  WIDTH <- zscore * sqrt(sum(estimate * (1 - estimate)/n)) + 0 * sum(1/n)
  conf.int <- c(max(DELTA - WIDTH, -1), min(DELTA + WIDTH, 1))

  model_coef <- (estimate[1] - estimate[2]) / estimate[2]

  prop_results <-
    data.frame(
      test_value = test_rate,
      ctrl_value = ctrl_rate,
      test_lift = model_coef,
      conf_int_low = conf.int[1]/estimate[2],
      conf_int_hi = conf.int[2]/estimate[2],
      row.names = NULL
    )

> prop_results
  test_value ctrl_value  test_lift conf_int_low conf_int_hi
1 0.03136381  0.0369963 -0.1522447   -0.2194746 -0.08501468

Ported to Java (using Apache Commons Math):

import org.apache.commons.math.MathException;
import org.apache.commons.math.distribution.NormalDistribution;
import org.apache.commons.math.distribution.NormalDistributionImpl;
...

long test_numerator = 1154;
long test_denominator = 36794;
long ctrl_numerator = 1749;
long ctrl_denominator = 47275;

double confidence = .95;

double test_rate = new Long(test_numerator).doubleValue()/ new Long(test_denominator).doubleValue();
double test_std = test_rate*(1-test_rate)/ new Long(test_denominator).doubleValue();

double ctrl_rate = new Long(ctrl_numerator).doubleValue()/ new Long(ctrl_denominator).doubleValue();
double ctrl_std = ctrl_rate*(1-ctrl_rate)/ctrl_denominator;

double delta = test_rate - ctrl_rate;
double test_lift = delta/ctrl_rate;
double conf_plus_one = confidence + 1;

try {
	NormalDistribution n = new NormalDistributionImpl();
	double qwidth = n.inverseCumulativeProbability(conf_plus_one/2);
	double width = qwidth*Math.sqrt(ctrl_std+test_std);
	double conf_int_low = Math.max(delta - width,-1)/ctrl_rate;
	double conf_int_hi = Math.min(delta + width,1)/ctrl_rate;

} catch (IllegalArgumentException e) {
	e.printStackTrace();
} catch (MathException e) {
	e.printStackTrace();
}

...

And Postgres SQL (not for the faint of heart!):

\set test_numerator 1154::float
\set test_denominator 36794::float
\set ctrl_numerator 1749::float
\set ctrl_denominator 47275::float

select
  :test_numerator/:test_denominator as test_value,
  :ctrl_numerator/:ctrl_denominator as ctrl_value,
  ((:test_numerator/:test_denominator) - (:ctrl_numerator/:ctrl_denominator))/
  (:ctrl_numerator/:ctrl_denominator) as test_lift,
  greatest(
    (:test_numerator/:test_denominator) - (:ctrl_numerator/:ctrl_denominator)
      - (
        -- z-score for 95% CI
        1.959964*sqrt(
          (
            (:test_numerator/:test_denominator)
              *(1-(:test_numerator/:test_denominator))/
                :test_denominator
          ) + (
            (:ctrl_numerator/:ctrl_denominator)
              *(1-(:ctrl_numerator/:ctrl_denominator))/
                :ctrl_denominator
          )
        )
      )
  ,-1)/(:ctrl_numerator/:ctrl_denominator) as conf_int_low_95,
  least(
    (:test_numerator/:test_denominator) - (:ctrl_numerator/:ctrl_denominator)
      + (
        -- z-score for 95% CI
        1.959964*sqrt(
          (
            (:test_numerator/:test_denominator)
              *(1-(:test_numerator/:test_denominator))/
                :test_denominator
          ) + (
            (:ctrl_numerator/:ctrl_denominator)
              *(1-(:ctrl_numerator/:ctrl_denominator))/
                :ctrl_denominator
          )
        )
      )
  ,1)/(:ctrl_numerator/:ctrl_denominator) as conf_int_hi_95;

The Gilt Tech shirts are here! If you love bacon...

giltquinn — Fri, 20 Jan 2012 11:35:30 -0500

The Gilt Tech shirts are here!

If you love bacon spearing narwhals, we might have the perfect shirt for you.

See all Lauren’s awesome designs on the Gilt Tech Shirt Store.

NoSQL in the Real World

giltquinn — Wed, 18 Jan 2012 14:01:00 -0500

For the last couple of years the interwebs have been abuzz with discussion on the relative merits of NoSQL database technology, but how has it performed in real world deployments?

Please join us for the second of the Gilt technical meetups to hear prominent local technology companies present on their experiences using Redis, Cassandra, MongoDB, CouchDB and Riak in production environments.

The event takes place on Wednesday, February 8th at 6pm. It’s free of charge on a first come first served basis. We expect it to sell out so reserve your spot early!

See details and sign up at http://nosql-tech-talk.eventbrite.com

Photo

mwunsch — Tue, 17 Jan 2012 10:19:55 -0500

My Time at Gilt

Mon, 12 Dec 2011 16:56:21 -0500

I arrived at 2 Park Ave on the first day of my internship not quite sure what I was getting myself into. I had just finished my sophomore year at Dartmouth and was looking forward to my first time living in New York City. My sophomore year was packed with computer science. I had taken the notorious software design and algorithm design courses, I was doing research with the chair of the Dartmouth computer science department, and now, I thought as I exited the elevator on the fourth floor, surely I was ready for the industry.

My first experience working with the Distribution team was a two day long architecture plan-a-thon for the new messaging system at Gilt, a system that facilitates seamless communication and synchronization between Gilt admin, distribution centers, and shipping companies. As the second day drew to a close, my boss, Chris Hazlett, began to explain my role in this slick new messaging system that we were designing. The start of that conversation went something like this:

Chris: “We’d like you to put together a web app with the spring mvc In scala that makes ajax requests to a web server serving search results from a lucene search engine which you’ll build and format the response with css and javascript,”

Me: “Um, yeah… What’s a web server?”

Everyone: “Oh boy.”

I had a lot to learn. We spent the next half hour going over a lot of the concepts I was missing. “web servers”, “model-view-controller”, “ajax”, and more. Often these words would rest on a whole stack of other important concepts that I also did not yet understand.

“What’s ajax?”

“Oh, well its just the process of making asynchronous requests to a separate web server to dynamically fill in content on a webpage. Don’t worry, you’ll just use jquery for that.”

“What’s jquery?”

“…a javascript library.”

“ What’s javascript?”

“…”

And so on.

My first few weeks at Gilt, I was very frustrated. How could my computer science program have left me unprepared for industry? Granted, my background is focused more on theory than systems. I do research in algorithm design and until I began to learn languages outside of my classes, my strongest languages were Haskell and C. But still, regardless of my theoretical focus, I was a computer science student, damn it. What was I missing?

It gradually dawned on me that most of my frustrations could be reduced to two missing links in my CS education: object-orientation and the web. These are two topics that are not strongly emphasized in traditional computer science programs. I resented the fact that there were no web development topics covered in any classes at Dartmouth, and that instead I had been exposed to what now seemed to be esoteric, dated concepts. The hours spent in gdb sifting through hundreds of lines of C to fix elusive seg faults and memory leaks now seemed futile now. The mind bending recursion, pointers, the linked lists, the hash tables, the mallocs, reallocs, the frees, ALL FOR WHAT! After briefly wallowing in my sorrow, my boss reminded me that the whole reason I was here was to learn. So I started to learn.

I spent my first weeks at Gilt learning new tools. My browser was constantly filled with tabs on scala, git, lucene, css, and jquery. The complexity of the problem assigned to me also became clearer. I was being asked to build the full stack of a web app. This meant writing everything from the css and javascript for the frontend, to the complex indexing and searching backend that would process and search new messages in real-time as they flowed through our messaging system. Okay, fine.

In reality, once I overcame that initial shock and the task of setting up my development environment (surely a twisted form of tech hazing), the general learning process was much more painless than I had anticipated. Most of the tools I was using were well documented, and when in doubt I had the most thorough and accessible form of documentation around: people! I consulted frequently with folks in all different groups, front end, algorithms, distribution, etc. As I learned the tools, my team was making fantastic headway on the new messaging system. We moved into a small conference room to make collaboration and communication easier. Danger Zone by Kenny Loggins became the natural theme of our industrious lair as we moved steadily towards the completion of our first iteration on the project.

Around this time, about halfway through my internship, the focus of my work in the office shifted from learning to building. As I finally understood how tools like Lucene, Jquery, and Scala worked and how all the pieces of a full web-app stack fit together, I began to feel more at ease and developed much more efficiently. I quickly made headway on the new user interface and the actor-based search engine library.

I also started to meet others outside of the tech team. Settlers of Catan frequently occupied my Thursdays, volleyball occupied my Mondays, and occasionally there would be, ahem, modest company gatherings held in abandoned four-story tall synagogues. The Friday of my second to last week with the company, I was able to present my final project to the rest of the tech team during the 5@4, which seemed well received. All in all, it was clear that my work was going to have a lasting impact, and would make the distribution team’s job faster and easier.

As my internship draws to a close, its now quite clear to me that the frustration I initially felt towards my computer science background was naïve. A Computer Science degree is not supposed to be a vocational degree. Computer Science is truly a science, and the point of studying it is to gain exposure to and mastery of the foundations. Once you know how to think like a computer scientist, learning new tools in the field comes quite naturally. That’s the idea, and thankfully, it works.

Gilt APIs

fabstractum — Mon, 05 Dec 2011 06:48:00 -0500

Over the next few months, Gilt will be exposing its data model via a new suite of public APIs. We’re excited to make our data available to a broader audience of hackers, and we can’t wait to see what people do with it.

Our first API is a set of Atom Pub feeds, which let you see our active and upcoming sales, either en masse, or store by store.

For example, https://api.gilt.com/v1/sales/active.atom shows all the active sales on gilt.com, or you can hit https://api.gilt.com/v1/sales/upcoming.atom to see what sales are coming soon.

You can see just the active or upcoming sales for some of our stores as well, for example:

All our available feeds can be concisely written using a regex:

https://api.gilt.com/v1/sales(/men|/women|/kids|/home)?/(active|upcoming).atom

We’ll be rolling out more details soon; this is our first toe in the water.

Enjoy!

- Gilt API Team

Movember at GILT. Here at GILT we care about each other, there...

spanmess — Mon, 28 Nov 2011 22:44:07 -0500

Movember at GILT.

Here at GILT we care about each other, there for we have, those who can been growing mustaches for the month of Movember. Why? To support the fight against cancer.

We can all change our faces, by growing mustaches, so let’s change the results of cancer. 1 in 2 men and 1 in 3 women will be diagnosed with cancer in their lifetime, so let’s change the result of this.

You can make your donation here, http://mobro.co/gilt

Gilt Home Launch

giltquinn — Tue, 25 Oct 2011 09:02:00 -0400