Gilt Technologie

C Level Engineering Symmetry

giltquinn — Thu, 16 May 2013 08:19:30 -0400

C Level Engineering Symmetry

Monitoring network traffic and service chatter with Boundary

sbailliez-gilt — Tue, 14 May 2013 12:32:18 -0400

We recently published a case study with Boundary regarding how we, at Gilt Groupe, are using their product and I wanted to give some additional details concerning our decision process, what we were looking for, what we looked at and why we decided that going to Boundary was the best choice for us moving forward.

Gilt Groupe’s architecture is now very much a case of micro-service architecture. We have hundreds of JVM-based HTTP services interacting with each others or with backend systems such as PostgreSQL, MongoDB, RabbitMQ, Kafka, Zookeeper, and many more third-party solutions over various data interchange formats and protocols.

A few months ago, we felt we needed to get more insight into the detailed amount of traffic that was going in and out of every service or backend system. When various teams are working on new features that require more communication patterns and data exchange, it starts to be difficult to do capacity planning when you don’t know where you are.

Moreover, in our experience, we have seen that most features generally go from a normal usage pattern for months to a sudden very large adoption by our business operations. The amount of data can suddenly grow 1 to 2 orders of magnitude, which does not generally go without its own set of challenges.

To get better insight into the amount of data exchanged, we started the effort to monitor the data transferred out of our HTTP services (we use Jetty) using the excellent Metrics library from Coda Hale. This can be trivially done extending the existing Metrics InstrumentedHandler for Jetty:


public class CustomInstrumentedHandler extends InstrumentedHandler { 

  private final Meter bytesTransferred = Metrics.newMeter(handler.getClass(), "bytes-transferred", "responses", TimeUnit.SECONDS);
  
  private final Histogram bytesResponse = Metrics.newHistogram(handler.getClass(), "bytes-responses"); 
  
  // constructor omitted for blog readability
  @Override
  public void handle(String target, Request request, HttpServletRequest httpRequest, HttpServletResponse httpResponse) throws IOException, ServletException {
    final AsyncContinuation continuation = request.getAsyncContinuation();
    try {
      super.handle(target, request, httpRequest, httpResponse);
    } finally {
      if (continuation.isInitial()) {
        long count = request.getResponse().getContentCount();
        bytesResponse.update(count);
        bytesTransferred.mark(count);
      }
    }
  }
}

The clients effort would be a bit more challenging however. In our JVM-based services we end using a menagerie of HTTP clients: AsyncHTTPClient with Netty 3.x provider, Apache HttpComponents 4.x, Apache Commons HTTPClient 3.x, and the venerable JDK HttpURLConnection.

This is the reality of having to deal with various third-party integration, it makes things more complicated than we would like to, but it can be a bit annoying to rewrite or extend some existing SDKs to try to use one and only one HTTP client across the platform (especially when non-extensible, or worse, closed-source).

The immediate problem faced is effectively how to instrument *all* those clients.

AsyncHTTPClient can be done easily using a RequestFilter and an AsyncHandler. The code would be something similar to the snippet below. There is not much overhead of doing it as you just need to count chunk size as they the HttpResponseBodyPart objects are received.

 public class InstrumentedAsyncHttpClientRequestFilter implements RequestFilter { 
  private final Meter bytesTransferred;
  private final Histogram bytesResponses;

  // ... initialization omitted for readability

  public FilterContext filter(FilterContext ctx) throws FilterException { 
    return new FilterContext.FilterContextBuilder(ctx) .asyncHandler(new MetricsAsyncHandler(ctx.getRequest(), ctx.getAsyncHandler())) .build(); 
  } 

 public class MetricsAsyncHandler implements AsyncHandler {
  private AsyncHandler delegate;
  private long totalBytesTransferred = 0;

  // ... initialization omitted for readability

  public STATE onBodyPartReceived(HttpResponseBodyPart bodyPart) throws Exception {
    long bytes = bodyPart.getBodyPartBytes().length;
    totalBytesTransferred += bytes;
    metrics.bytesTransferred.mark(bytes);
    return delegate.onBodyPartReceived(bodyPart);
  }

  public T onCompleted() throws Exception {
    T o = delegate.onCompleted();
    metrics.bytesResponses.update(totalBytesTransferred);
    return o;
  }
 }
}

Note that we tend to give a name to each service client which would map to a Metrics scope, which makes it useful to distinguish metrics between each client (some services use a dozen of clients).

For all the others clients, it is a bit more intrusive to be practical. And it doesn’t address how to monitor the traffic in/out going directly through the Socket api like for Zookeeper, Play Framework (Netty server), MongoDB and JDBC drivers, etc…

Another solution would be to write a JVM Java Agent via the java.lang.Instrument API to instrument some well known libraries (NewRelic uses a similar technique, but doesn’t track traffic). While it may looked like the less intrusive solution, it is also a fairly significant undertaking to develop instrumentations for several third-party libraries which you have to maintain over time.

Also, knowing we were looking with an interested eye to add systems such as Riak, Redis and possibly some various asynchronous drivers and having to deal with multiple versions of Scala… this was a cool project to work on technically, but maybe not excessively practical.

What we needed was something similar to nethogs minus the text interface. A tool capable of grouping the bandwidth by process, but ideally it would have some features also found in Wireshark.

We did not find anything matching those requirements.

Until a week or two later. We had Cliff Moon, Co-Founder and CTO of Boundary, visiting our New-York office to present Boundary and do a Tech Talk on Distributed Systems (which we blogged about).

We installed Boundary on some our servers to get a better idea. This was truly a revelation. The installation was painless with just a single command and as soon as the agent was up, it started to report data to the dashboard within the next second.

Each of the line represent the traffic volume happening on a given port/protocol across all nodes at a 1 second resolution. Traffic can easily be broken down. For example you have the ability to group servers, either manually or dynamically using pattern matching which makes it easy to segment your front-end from your backend machines and see traffic flowing between those groups (this is where a descriptive naming policy for your machines comes handy).

You can further segment your traffic by port / protocol. For example TCP 5432 would be the traffic to/from PostgreSQL. You can then easily analyze the traffic that is going from your backend machines (or a subset of those) to your PostgreSQL. Same thing could be done to know the chatter around our messaging infrastructure on RabbitMQ.

A lot more details on how all of this can be done is visible on a Youtube video ‘Isolate your traffic with filters and conversations’.

There is a shortcoming currently for us where we are effectively losing a bit of visibility in our conversations. For instance, traffic to our services is always going through a set of dedicated service load balancers. For example we reach to it via a canonical url such as http://svc-product and the load balancer will balance between node1:7501, node2:7501, node3:7501. It means traffic from the caller to/from the load balancer happens on port 80 while the traffic to/from the load balancer to the callee is on port 7501.

caller ← port 80 → svc-lb ← port 7501 → callee

Which means that the traffic flowing on port 80 is basically the aggregate of all service traffic and that we cannot see the traffic directly from caller to callee, but only the aggregate from caller to svc-lb and from svc-lb to callee.

This is something that may be alleviated a bit in the future as we are thinking about removing the load balancer and having applications doing the load balancing themselves using information from Zookeeper.

Boundary settings on the dashboard can be driven entirely from their REST API, which provides the added convenience of being able to integrate with your own configuration management system such as Puppet or Chef and a set of backend applications which may contain metadata about your environment.

The REST API is useful to define application aliases which gives names to a protocol:port (eg: ‘svc-product’ for TCP 7501), send deployment events or integrate with other systems (it can subscribe to NewRelic events via RSS).

We have only scratched the surface of Boundary so far and we are very excited about the direction it is taking and what is being developed. It has already proved extremely useful in identifying traffic volume and patterns occurring between services and databases. Something that would have required a lot more tedious investigative work can basically be done now in a few minutes and with much more flexibility than we could initially imagine and with no direct investment.

I hope that this (long) blog post will be helpful to some people who are facing the same challenges of not having enough visibility in their network traffic. If however you know of any interesting tool in that space, feel free to drop a note.

On a slightly unrelated note, we are also users of a nice library from Boundary called Ordasity. It is a great way to distribute workload across nodes via Zookeeper. It was brought to our attention during Scott Andreas’s tech talk at Gilt Groupe (another one !), and it might be the topic of another blog post.

— stephane

Mobile Web How To: Development Environment Toolkit

gregorymazurek — Wed, 08 May 2013 16:26:00 -0400

While working on Gilt Mobile Web, these are our development tools of choice to get the job done:

Browsers & Emulators

Android emulator: http://developer.android.com/sdk/index.html

XCode iOS Simulator: https://developer.apple.com/devcenter/ios/index.action

Chrome Canary + mobile settings: https://www.google.com/intl/en/chrome/browser/canary.html

Page Speeds

HTTP proxy/monitor and bandwidth throttler: http://www.charlesproxy.com/

PageSpeed Insights: https://developers.google.com/speed/docs/insights/using_chrome

Device Testing

User agent strings: http://youruseragent.info/commonua.en.htm

Feature compatibility checks: http://caniuse.com/

Android debug bridge for inspecting your Android device: http://developer.android.com/tools/help/adb.html

Adobe inspect for inspecting Android Browser: http://html.adobe.com/edge/inspect/

iOS 4.2.1 to 6.1 iPods and iPhones

Android 2.1 to 4.2 phones and tablets

Usability Testing

Reflector: http://www.reflectorapp.com/

So, what tools do you use?

Mobile Web How To: Proxy Local Environment To Devices

gregmazurek — Tue, 07 May 2013 17:54:30 -0400

When you develop a front end experience for the full screen, the feedback loop between code and test is relatively fast. You code. You refresh your browser. Repeat. When you develop a front end experience for mobile devices, this can become a little cumbersome because the feedback loop can sometimes feel much slower. There are emulators for your machine and browser sizing/user-agent strategies that you can employ to make your development process more efficient. But in the end, you’re going to want to test on your devices.

In this post, I’ll explain how you can proxy your local development (localhost) to both your Android and iOS devices. When you do this, you’ll be able to code, refresh, repeat a lot faster and more efficiently.

Charles Proxy

To get started, you need a strategy to manage your HTTP proxy. I use and recommend using Charles Proxy (http://www.charlesproxy.com/) but there are certainly other alternatives. Charles Proxy is very powerful but I won’t be going into much detail about it here. Instead, we want to set up a port that we can HTTP proxy to.

If you click on PROXY, then PROXY SETTINGS, you’ll see a menu where you can enter in a port that you would like to proxy to. I’ve chosen 8888.

Charles Proxy is going to look for incoming connections on this port. When you connect iOS or Android to this port, you will see that Charles Proxy will ask you to allow or deny this connection attempt:

Now, let’s connect our devices.

iOS

In iOS, navigate to your WiFi menu and then tap into your connected WiFi. At the bottom of this screen, tap on MANUAL under HTTP Proxy. Where you see computer_ip_address_here, enter your computer’s ip address and where you see 8888, enter the port that you set up on Charles Proxy.

That’s about it. On your device, open localhost:1234 or whatever in a browser and then Charles Proxy will ask you to allow or deny. You can now start coding on your machine and refreshing on your iOS device. You’re done.

Android

On Android, tap SETTINGS and then Wi-Fi. Next, tap and hold on the WiFi network that you are currently connected to. On the following menu, tap on MODIFY NETWORK CONFIG.

You next see a menu to manage your network config. Scroll to the bottom of this modal and tap on SHOW ADVANCED OPTIONS.

In the menu options that appear, you will see configuration settings similar to those found in iOS. Where you see computer_ip_address_here, enter your computer’s ip address. Where you see 8888, select the port you set up with Charles Proxy.

And, that’s about it. On your device, open localhost:1234 or whatever in a browser and then Charles Proxy will ask you to allow or deny. You can now start coding on your machine and refreshing on your iOS device. You’re done.

Today in Pictures

giltquinn — Tue, 07 May 2013 17:15:00 -0400

Today in Pictures

Mobile Web How-To: Detect Samsung S4 Device On Your Mac

gregorymazurek — Mon, 06 May 2013 12:48:01 -0400

This hasn’t been very well documented on the internet just yet so I thought I’d share how you need to connect your Samsung S4 device to a Mac. When you connect this device to your Mac, it (most likely at the time of this writing) will not be recognized.

To have your Mac detect your Samsung S4, you have to do some magic on your device. Click on into SETTINGS and then click on ABOUT DEVICE. If you scroll to the bottom, you will see a line entitled BUILD NUMBER.

Tap on the build number 7 times. I’m not kidding.

When you are now “a developer,” you’ll see a new menu option in SETTINGS -> MORE entitled DEVELOPER OPTIONS. In this option, you need to check USB DEBUGGING.

Then when you next connect your device to your computer, you’ll see your device:

Mobile Web How-To: Inspect Elements On Android's Internet Browser

gregorymazurek — Fri, 03 May 2013 15:51:34 -0400

I’m building Gilt’s new Android app and a good portion of the website is an Android WebView. As you may or may not know, this WebView uses the default Android Internet Browser to render webpages. You probably know this app best by its logo in the lower right hand corner of this screenshot:

When you use Google Chrome on the Android device, inspection is very straightforward — I’ll cover this in a later post. But for Android Internet Browser, there is not to my knowledge a good way to inspect and manipulate the DOM.

I needed to inspect the DOM because I inherited a JavaScript file that allows us to mimic scrolling events on mobile devices via webpages. The scrolling library works as expected in all other browsers on all other devices. So, I needed to better understand what was happening in the Android Internet Browser.

The tool that bridged my device to an inspector tool is Adobe Inspect. To get going, you have to install 3 components:

1. Adobe Inspect on your computer: http://html.adobe.com/edge/inspect/

2. Google Chrome Extension Adobe Inspect: https://chrome.google.com/webstore/detail/adobe-edge-inspect/ijoeapleklopieoejahbpdnhkjjgddem?hl=en

3. Google Play Store Adobe Inspect: https://play.google.com/store/apps/details?id=com.adobe.shadow.android&hl=en

Once you have installed everything, connect your Android device to your computer and make sure you’re on the same WiFi. On your Android Device, open Adobe Inspect and click the plus sign in the upper-right hand corner to get to this screen:

Get your ip address from your computer and enter it in Adobe Inspect on your Android Device. If you skip ahead, you can find your ip address on your computer by opening Google Chrome and clicking on the Adobe Inspect icon in the nav bar — you’ll see it there as well. After you input your ip address into the Android device, you’ll receive this screen:

Now, open Google Chrome and open the url of your choice with the Adobe Inspect extension enabled. In the upper right hand corner, you’ll see the Adobe Inspect icon with a green plus sign.

Click on this icon to reveal a menu that displays your computer, your IP address, and your device name.

Enter the Passcode you received from your Android device.

Now that you’re connected, click on this button next to your device name:

Then, a new window should open that looks like this:

You can see that this is a standard Google Chrome inspector with the name of your device and the url that is currently being inspected. Click on this link and then click on Elements.

Here, you have a standard inspection workflow similar to what you would use for the full screen experience. You can use the console and other features in a more limited manner to what you would use on the full screen experience. And, Adobe Inspect will highlight what DOM elements are being inspected on the device:

There is much more you can do but, hopefully, you’re now set up to debug the Android Internet Browser (not that you needed to debug anything in the first place).

Offsite

giltquinn — Wed, 01 May 2013 09:19:00 -0400

A few of us took some time to get to know each other a little better, hug trees and talk shop.

ade-trenaman: Erlang, Distributed Systems and Sierra Nevada...

ade-trenaman — Tue, 30 Apr 2013 20:28:19 -0400

ade-trenaman:

Erlang, Distributed Systems and Sierra Nevada Pale Ale

Found this photo on my phone - a real moment in time at Gilt New York! Steve Vinoski, who worked with Gregor and I back in the days of the seminal Irish tech startup IONA Technologies, came to Gilt one cold February night to give a talk on how Riak is implemented in Erlang - it’s a great talk, and a super introduction to Erlang. From left to right: Eric Bowman, Steve Vinoski, Gregor Heine, me, and Mike Bryzek. Too. Much. Fun.

Ade Trenaman: Why Java 8 doesn't rock my Scala

ade-trenaman — Thu, 25 Apr 2013 10:55:04 -0400

Ade Trenaman: Why Java 8 doesn't rock my Scala:

ade-trenaman:

I was looking through an article on Java 8, . We’re using Scala heavily at Gilt and I was toying with the heretical notion that Java 8 might create a compelling reason to go back to Java. Sacrilige! I am of course biased in this matter as I’ve really enjoyed the last two years of Scala coding at…

Gilt Mobile Web Nav Redesigned

gregmazurek — Wed, 24 Apr 2013 11:00:37 -0400

Hello!

Yesterday (April 23, 2013), I sent to production the third phase of the Gilt Mobile Web (m.gilt.com) redesign. In this phase, I updated the primary and secondary nav on the mobile web so that there is (hopefully!) a much better user experience. And like the first and second phase of the redesign, the goal with the m.gilt.com nav redesign was to incorporate learnings taken from the Gilt iPhone App experience.

Here are some before and after screenshots:

On the Sales Listing Page, you’ll see that I moved the stores menu from above the primary nav to below it. This gives the overall navigation a hierarchy that was not present earlier. You’ll also notice that the nav feels less busy because the buttons in the primary nav are more concise.

In the Product Listing Page, you’ll see that the back button was changed and that the title of the sale occupies the secondary nav. This allows the user to easily know in which store she is. In the future, the secondary nav on the product listing page will disappear as the user scrolls down.

On the Product Detail Page, I tightened the primary nav a little more and there is still no secondary nav.

On the checkout page, I changed the styling of the Cart and Submit button. The colors now match the style that persists throughout the entire Gilt Mobile Web experience. And, the lock is gone. Our site is still secure regardless whether the lock is there or not.

Let me know what you think!

Greg

Up & Running with Play 2 Last night, Gilt hosted the second...

mwunsch — Thu, 18 Apr 2013 17:02:54 -0400

Up & Running with Play 2

Last night, Gilt hosted the second Play NYC meetup, after nearly a year since the first. This meetup was dedicated to introducing people to the Play Framework.

I gave a presentation that briefly touched on the reasons why Gilt chose the Play Framework for some of its recent projects and then jumped right into code. Over the course of the talk, I walked through constructing a Play app with Scala connecting to Gilt’s API. Starting with some functional tests, we covered:

routing,
controllers,
Play Actions,
making a request to a web service,
Play’s configuration
Futures and asynchronous behavior,
unmarshaling JSON,
Play’s templating language,
compiling Assets,

And finally, and most importantly, deploying our application to production (with Heroku). Along the way we touched on some tenets of functional programming¹ and Scala’s powerful for comprehensions. We covered a lot of ground and in the end only ended up with one route: listing active sales by store.

The slides for the talk are available on SpeakerDeck, the application built is currently running in production, and the application code is available on GitHub for anyone to fork and experiment on.

I had a blast. Our next meetup isn’t scheduled yet, but we definitely won’t allow such a long delay between meetups again. Join the group to be the first to hear about upcoming events. Thank you to everybody who came out. I am looking forward to the next one.

— Mark

“What’s a monad?” ↩

Working with Time Series Data

giltquinn — Mon, 08 Apr 2013 12:08:00 -0400

Life in Gilt Tech involves much lively discussion on KPIs, consequently we spend a lot of time looking at time series data.

We use a variety of tools to manipulate and present this type of data, including R, a language and environment for statistical computing and graphics.

To help us through the process of standardizing the way we represent our KPIs we have developed a simple, R based open-source utility for easily representing time series data originating from a variety of sources.

If you think you might find this useful, or you’d like to contribute, check out the time series project on github, and the examples below.

Example: a thumbnail showing acme revenue by time:

plot_time_series --width=500 --height=400 --x_spacing="2 months" --csv_filename=test_data/acme_revenue.csv --point_color=gray80 --sunday_point_color=gray80  --remove_outliers --smoothness=0.8 --y_line="35000:Target" --y_prefix="$"

Example: the same revenue data shown in more detail

plot_time_series --width=1200 --height=700 --x_spacing=week --csv_filename=test_data/acme_revenue.csv --title="Acme Revenue: %s to %s" --remove_outliers --y_line="25000:Target for 2012:gold3!36000:Target for 2013:gold2!42000:Target for 2014:gold1" --y_prefix="$"

More plot time series examples and documentation …

Gilt Mobile Web Redesigned - 10 Views Compared

gregmazurek — Fri, 05 Apr 2013 15:54:16 -0400

Over the last three months, we overhauled the front end (jsp, html, handlebars, less/css, javascript, zepto) for Gilt’s mobile web experience (http://m.gilt.com). The redesign was inspired by learnings acquired from our iPhone App and the design is meant to replicate a lot of those features.

Here are 10 side by side comparisons with notes inline and at the bottom. Let us know what you think!

In the previous design, there was an assumption that the Gilt shopper uses the mobile experience quickly and wants to see as many sales as possible in a short amount of space and time. But according to our iPhone App results, our users want to see larger pictures. Having two sales on the first view compared to four has resulted in increased sales, largely due to our amazing imagery.

We dropped the black background on the product details because most of our imagery adopted a white background.

A nice feature of the redesign is that the add to cart button is visible on the bottom while scrolling on the product detail page. If you are shopping on m.gilt.com and want to purchase an item, you should have the add to cart accessible at all times. But, this doesn’t always work on all devices.

We created a more streamlined view for our sign in and registration pages by simplifying the experience.

Our cart page features a similar button to what existed on the product detail page. Again, we are trying to make it as easy as possible to shop on a mobile device.

The checkout experience mimics the iPhone App experience while trying to keep a style that is more generic so that it looks great on Android as well.

While scrolling to the bottom of the checkout page, the submit order button stays fixed. In our older version, the user had to scroll to the button to checkout.

We are reusing elements in our account page that we used in the checkout flow.

Our footer introduces more spacing as well as elements that were previously found in the checkout and account flow.

We have an amazing customer service team.

Here are 5 learnings from redesigning the Gilt Mobile Web experience.

1. Visual Components: Abstract away visual components that can be reused throughout the mobile experience. For example, the buttons in the footer are also used in the account page as well as in checkout. We only need to supply text and an optional image to create one.

2. Cross-Browser Compatibility: Handle browser-specific issues like fixed-positioning on a per-device basis. The fixed “submit button” appears at the bottom of the screen on the product detail page, cart, and checkout page for modern browsers. But for early versions of Android, for example, the buttons appear inline.

3. Iterating: Gilt Mobile Web is a complete experience in that it includes everything from the sales listing page through account and checkout. When trying to roll releases out to production, we searched for places where the old and new designs could coexist for a limited period of time. With this strategy, we didn’t have to redesign everything at once.

4. Cross-Device Compatibility: We tested m.gilt.com against several android and apple devices (as well as ie6 and ie7) to make sure it looked good on many different screen sizes. In many instances, DOM element sizes are determined as a percentage of the viewable screen. But since the majority of our users are accessing us from Safari on iPhone, development starts on the iOS Simulator.

5. If it doesn’t work, roll it back: I deployed our second phase to production and immediately upon doing so realized that “sold out” hover states were not… hovering. It was something that we missed and instead of making a quick fix, we rolled it back and spent a little more time testing. We pushed to production again a few days later with bug fixes in place and confidence that the experience was stable. Sometimes, we can’t catch it all on staging environments.

26/3/13: Today's Noon Outage - and what we're doing to make sure This Never Happens Again.

ade-trenaman — Tue, 26 Mar 2013 16:28:00 -0400

At Gilt we try to move as fast as we can getting code - be it fixes or awesome new features - to production as quickly and safely as possible. Sometimes we make mistakes, and, today was such a day. Around noon, a commit on one of our flagship applications ran riot: allocating native threads; consuming memory and CPU; and bringing down all other applications collocated on the same set of servers. Our customers were affected and for this the tech team at Gilt are truly sorry. There have been a ton of tweets from concerned members, and we were keen to explain what went wrong.

Some interesting things arose in the aftermath of this outage (we take all our outages very, very seriously). First and foremost, this was largely a people problem: our process around code-review and performance testing failed - a commit that was not fully peer-reviewed was moved to production, and deployed -after- our morning performance run. Under normal load, everything looked ‘just fine’ - under our noon rush, everything fell apart. I estimate there’s about a 60% chance that a good code-review would have caught the issue - however, I’m 99.999% confident that our rigorous load tests would have caught the issue. We use Gerrit for code review, and have automated load tests every morning; this was a late change for which we were hoping to gather some real life data related to load - ironically to debug a smaller performance issue we’ve been seeing in one of our applications - and we rushed the change.

A positive we can take out of this as a tech organization is that we already have a number of initiatives in progress at Gilt that will prevent this kind of thing from happening. The more we use code-review, the more we see opportunities to automate the veracity of any release: you can imagine a script that says “this release contains a commit that was not reviewed and was +2’d by the author - abort!”. From a performance perspective, we’re working on incorporating performance testing into a continuous integration system we call ‘ION Cannon’; again, any release would be performance tested and rolled back automatically. These are areas where we can take human error out of the process, and I know that I (as the engineer who didn’t run the load test) am keen that we double our efforts in this regard.

One of the other things we’re doing is making all of the applications that run gilt.com smaller and more isolated, an initiative we call ‘LOSA’ (Lots of Small Apps). LOSA is the mantra that lets us break up some of our large web applications into smaller chunks. The outcome? Stronger ownership of each part of gilt.com by the engineering teams; simpler code, with better sharing of assets and commons functionality; and, better isolation - in the event that one of our applications serving a part of gilt.com goes awry, other apps should not be impacted.

Today we made a mistake - and we feel pretty bad about it. We feel good though about what we’re doing to make gilt.com better.

Intro to AngularJS + Gilt API

ffglt — Mon, 18 Mar 2013 10:55:16 -0400

Intro to AngularJS + Gilt API:

Brett Coffin explains how to build a super slick web application that shows off what you can do with the Gilt API.

SXSW 2013 Recap

parkd — Thu, 14 Mar 2013 11:43:18 -0400

SXSW 2013 Recap:

The Gilt UX team was at the SXSW Interactive conference this year, and we live blogged all of the sessions we went to. Click in to see our notes on Google’s innovation group and their Glass demo, the future of Foursquare and location, the emergence of tech in fashion, and more!

Scaling Agile at Gilt With teams, ingredients, initiatives and KPIs Spotify recently documented...

giltquinn — Thu, 07 Mar 2013 12:46:00 -0500

Scaling Agile at Gilt

With teams, ingredients, initiatives and KPIs

Spotify recently documented their progressive approach to scaling Agile development with a fairly large team. Gilt’s approach has many similarities, but since some Spotify best practices are Gilt anti-patterns, it’s worth a closer look.

The Gilt tech team is about 100 strong. Most of our development is done in Manhattan. We’ve also got a small team in Portland, Oregon and a larger one in Dublin, Ireland.

Initiatives

The cornerstone of the Gilt process is the initiative. We define an initiative as “a project we expect to work on for the foreseeable future.” Our foreseeable future is typically 3-12 months, depending on the area. An initiative might be to increase organic search traffic to the site.

We work on the order of 10 initiatives in parallel. The decision to green light an initiative depends on a number of factors. The most significant factor is a mathematical model of how we expect the initiative to perform, but we also consider softer factors like direct customer happiness and innovation.

Prioritizing a set of initiatives focuses the technology group on an initiative portfolio. This portfolio makes a clear statement on what is important, and indirectly, what is not. This has had a profound effect, halting our previously unending feature level prioritization discussions. This change has made not only our team happier and more productive, but also our stakeholders.

Unlike Spotify, we don’t maintain a roadmap document. Our roadmap is simply the sum of active initiatives. We revisit the set of prioritised initiatives every few months.

KPIs

Initiatives are defined in terms of the key performance indicators (KPIs) used to measure them. We define two types of KPIs. Strategic KPIs measure the success of the initiative’s strategy and last as long as the initiative. Tactical KPIs measure the performance of the initiative’s tactics and come and go as tactics evolve. All tactical KPIs must drive a strategic KPI.

For example, our SEO initiative might be defined by the strategic KPI: “Number of organic visitors in the last 30 days.” If current tactics involve stimulating inbound link creation a tactical KPI might be: “New inbound links in the last 30 days.” That tactical KPI would be abandoned when the initiative no longer employed that tactic, either because the team no longer thought they could get more out of that tactic or the team discovered a new tactic that could drive their strategic kpi faster.

We consider initiative mission statements an anti-pattern. “To be great at Organic search!” is simply a more subjective, and less useful denormalization of the strategic KPI.

Teams

We organize ourselves into teams (Squads in Spotify parlance) of around 4-10 cross functional members who sit together. Teams focus on a single initiative but may own others that are less active. Team members have a lot of input in selecting the initiatives their team works on.

Team dynamics take a while to form and stabilize. Consequently we favor long running teams, surviving generations of initiatives. However mobility between teams is encouraged and takes two forms. Teams self-organize to swap engineers periodically to manage skills deficits or unexpected workloads, in a practice affectionately called “horse trading.” When teams take on new initiatives it’s not unusual for an engineer with a particular interest to permanently switch teams to work on that initiative.

Unlike Spotify Squads, teams are not tied to Initiatives. We anticipate teams outliving initiatives, and being able to handle concurrent initiatives. This doesn’t preclude a team having a single initiative for it’s lifetime.

Like Spotify, we strive to make a team operate and feel like a startup. Our teams have a strong identity and are empowered and independent. Teams are self organizing and self prioritizing.

Teams select their own development methodology. Agile methodologies are strongly encouraged and most teams have backlogs, stand-ups, sprints and retrospectives. Some adopt more rigorous Agile methodologies.

Strong team code stewardship is encouraged. We use stewardship to describe an ownership model where one team owns repositories of code and takes responsibility to encourage, review and approve modifications made by other teams. Gerrit is used to facilitate the review process.

Unlike Spotify, teams have no product owners or business owners. Some teams include business partners. Ownership is considered an anti-pattern since it implies disempowerment of the team members who are not owners. KPIs and collaboration are king. Also unlike Spotify, we don’t appoint different individuals to the doing and prioritizing of work, believing these concepts are fundamentally indivisible. Prioritization happens by the team vigorously debating hypotheses and coming to agreement on the next hypotheses to test, factoring in both impact and feasibility.

Teams release early and often. “Minimum Viable Product” is a Gilt mantra.

Teams are encouraged to be highly analytical. We a/b test almost everything and discourage making decisions based on gut instinct. However, we respect intuition and encourage broad hypothesising to ensure we optimize towards global rather than local maximas.

Ingredients

Teams are organizationally flat and cross functional. To increase the likelihood they’ll execute successfully on their initiatives we use a palette of team ingredients as a guide to ensuring the team is appropriately staffed.

For a team to be successful at Gilt they need some combination of the ingredients: product visionary, coder, presenter, visual designer, business thinker, organizer, motivator, quality manager, coach, analyst, architect, writer, visual designer and ux specialist.

Rather than filling a narrow role, team members add a set of ingredients. This expectation both pushes and nurtures each team member. We don’t consider “It’s not my job to …” an acceptable way to start a sentence.

Depending on each team’s initiative, a different blend of these ingredients is optimal. One person might provide several ingredients, for example, a member who conducts tests, writes feature code and collects and analyses data.

Teams lacking key ingredients are assisted either by adding additional members or by external coaching.

Oversight

A bi-weekly check-in gives each team a regular opportunity to provide an update on their results and seek guidance on their current direction. In addition to the team, attendees to this meeting alternate between a panel of technical managers and a panel of senior executives from across the company.

All check-in meetings have the same agenda. (1) a presentation of KPIs and their history (2) a presentation of recent a/b tests or other recent data and the associated learnings and insights (3) the story: a quick overview of tactics for the next few weeks.

The prescriptiveness of this agenda is to encourage teams to go deep on the strategy and the data supporting it and avoid discussing feature and UX nuances. Unchecked, these discussions historically tended towards feature debate at the expense of discussing results.

Enabling Autonomy

To allow teams to operate as autonomously we encourage the decentralization of almost everything.

Decentralization minimizes the dependencies teams have on other teams or systems, allowing them to develop code and release it to production with minimal external interaction.

Recently we’ve made a large investment in the tools and automation to enable this decentralization. We’re well on our way to fully automated system provisioning and configuration.

We’ve made a similar investment in our application architecture, moving away from monoliths and towards lots of small independent applications and lots of small independent services. This in combination with our continuous delivery pipeline allows teams to take complete control over delivering software to production. Although we’ve made a big investment here, it’s a work in progress.

Extra-team Communication

In addition to the bi-weekly check-in meetings, we encourage the use of several other mechanisms for cross team communication and collaboration.

Tech Bytes: Monthly, each team sends out an email to a broad distribution, outlining the recent achievements of the team, framed by their KPIs.
Lightning Presentations: Teams self-organize to run a meeting with quick-fire presentations on notable pieces of technology or features that have been developed or deployed recently. Everyone is welcome at the meeting and pizza and beer serve as bait to attract a wide audience.
Hack sessions: techies are encouraged to periodically take a break from their initiatives and join forces with other teams to work on things they care deeply about. Projects run the gamut from architectural changes, tools, bug squashes, a/b test infrastructure to user facing features.
Stakeholder Communication: Teams self organize to communicate regularly with and get input from their various stakeholders.
Guilds: Gilt has a fairly loose implementation of the Guild concept, defined by Spotify as an “organic and wide-reaching ‘community of interest’, a group of people who want to share knowledge, tools, code, and practices.” We have fairly active groups that self organize around areas including front end engineering and architecture.

Does this Process Work and Will it Work for You?

Over the years we’ve used several different approaches for managing and organizing the Gilt development team. The one described above has been in active use for about 15 months.

The current approach embraces technology as a key differentiator for Gilt and serves to establish our technology department as a leadership organization rather than a service organization.

Being a leadership organization sounds great, but it’s not a panacea. It creates additional demands on teams, and requires them to have a deep understanding of our business and a strong point of view, clear vision and driving ambition. We need entrepreneurs to make this model succeed.

Not all technologists want to take this on. If you do, this approach might work for you.

Love the tech badge of honor stickers appearing around the...

giltquinn — Tue, 26 Feb 2013 09:10:00 -0500

Love the tech badge of honor stickers appearing around the office recently

Awesome Galactus Sticker design by Tim Flanagan

Gilt at the Fashion Hackathon

frutafresca — Wed, 20 Feb 2013 18:45:18 -0500

Hello there!

I’m Miguel Fajardo, an engineer at Gilt. I want to share a great experience I had with fellow Gilt engineers Jose Sanchez and Patrick Losco at the recent Fashion Hack during New York Fashion Week.

Hearst Corporation hosted the event in their New York City headquarters, the Hearst Tower. The building is easily identified by its diamond-patterned façade. Many other companies co-sponsored the event, including Microsoft, Spotify, Okta, HTC, and of course, Gilt.

The two-day event kicked off on Saturday morning in the auditorium where Hearst explained the rules of the competition. The goal was to build a web or mobile application that would interact with the APIs of some of the sponsoring companies, that included Gilt.

As one of the companies exposing the APIs to be used as part of the competition, we presented our developer portal and showed some examples to illustrate how to use the Gilt API. Anyone can query our systems to retrieve information about current or future sales, including what products will be included and a ton of info about them (price, current availablity, checkout URL, etc). If you are interested, you can find more info in our developer portal.

Here you can see the team once dinner was served at 7pm:

From left to right: Miguel, Jose, Patrick, and Dalia (she joined our team as application designer and front end developer).

Our idea was to use a city selected by the user and the current local weather to show trends in the form of relevant clothing and accessories from Hearst media. When the user chooses an image, the application recommends similar products available on Gilt. We built it using the Play framework with Java and Twitter Bootstrap.

By midnight, we had the retrieval of content from the APIs completed and started focusing on the front end. Time flies in this kind of event, where you are trying to design and build an entire application in such a short period of time!

We had all major functionality wired up by dawn, and then it was a matter of improving the user interface and optimizing performance. We improved the algorithm to select what images from Hearst to show and how to link these to the Gilt items on sale. We decided to fetch the weather at runtime, but cached other API calls that were taking up to a minute. The submission deadline was Sunday at 1pm and we made it just in time. There was no time to rest before all the teams presented their work.

The presentations took place on the 44th floor of the amazing Hearst Tower.

The views were stunning, especially with fresh snow from the previous day’s blizzard covering the city.

Our presentation went smoothly and we even appeared in a tweet from Hearst president David Carey.

In the end we got a special mention as one of the top 5 applications.

Jose was already asleep by the time we got to the prizes.

The Hearst Fashion Hack was a great experience. Not only did we learn a lot about the APIs, but we had a lot of fun together. You can view a demonstration of the application in this video.

Hope you enjoyed the post!