The Gilt technology organization. We make gilt.com work.

Gilt Tech

iOS Custom Gesture Recognizer

Paul Lee ios

As part of our Mobile team’s ongoing effort to introduce more Swift-based pages in the iOS app, I recently started working on a full screen gallery feature on our Product Detail Page (PDP). The feature is similar to the typical full screen photo gallery users will recognize from many other apps, including Apple’s Photo app and e-commerce applications featuring full screen view of product images.

Quick Way to Exit Full Screen View

In order to offer the best shopping experience, we wanted to make sure our customers can easily enter and exit the full screen image view from PDP. Entering full screen view is easy - customers simply tap anywhere in the photo area. For exiting full screen view, we wanted to let customers scroll vertically with a single finger to zoom-out and, if enough zoom-out level is reached, exit full screen. Performing double finger pinch-in gesture to zoom-out beyond threshold level will also close the full screen view but single-finger gesture is much easier and more natural while holding the phone in one hand. It also helps customers browse through products faster. This is crucial when every second counts for grabbing last minute inventory on Gilt’s flash sales!

Adding a New Gesture Recognizer

When customers are on full screen image view, they can scroll left or right to browse the next or previous image. That screen movement is handled through UIPanGestureRecognizer - one of the built-in gesture recognizers in iOS. Gesture recognizers provide (x,y) coordinates of users’ finger movement as a gesture is performed. New (x,y) coordinates are captured and passed to application every 16.7ms, or 60Hz.

To allow browsing to the left or right, PanGesture listens for x-coordinate change in a user’s finger movement and moves screen to the left or right accordingly. It ignores y-coordinate change. For example, if a finger moves diagonally to the left, it creates the same effect as moving horizontally to the left as change in y-coordinates are ignored.

In order to support scroll-down-to-zoom-out-and-exit function, we added a new gesture recognizer that ignores x-coordinate changes and only recognizes y-coordinate change. That is, if a finger moves diagonally to the left, it will zoom out the image and, once enough zoom-out is applied, exit the screen. The application will zoom out images gradually as the y-coordinate changes, ignoring all x-coordinate changes.

Conflict

We now have two gesture recognizers on the same screen that perform opposite work of each other. One listens to delta X and ignores delta Y. The other one listens to delta Y and ignores delta X. Given how dense iOS device screens are and how fast finger movement is captured, it is very unlikely that a user’s finger movement is perfectly horizontal or vertical. Thus, we assume that we will always see diagonal movement, which changes both x and y coordinates at the same time. If a finger moves diagonally, we cannot let both gesture recognizers be activated because we must perform only one of the two actions - scroll to next/previous image or scroll-to-zoom-out.

We know we have to pick one of the two recognizers very early on when gesture begins. That way, we know what to do with subsequent x, y coordinate changes. Interestingly, customers expect that too. They understand that if they start horizontally scrolling a page, the vertical change in their finger movement won’t have any effect while horizontal scroll is being performed. The same goes for scroll-down gesture. They expect that once their intention to scroll down is recognized, any change in horizontal movement during the rest of scroll would have no effect.

The question then is how do we read user’s intention very early in their finger movement and pick the right gesture recognizer to activate?

Solution

The answer can be found in the first two set of coordinates obtained. We obtain the very first two set of coordinates and if delta Y > delta X, then customers’ intention is to scroll down. If delta X > delta Y, the intention is to scroll to the left or right. These two set of coordinates are obtained within a fraction of a second as mentioned above and thus the distance covered on the screen will be extremely short. Yet, this is indeed how users tell us what their intention is.

To include such decision process as part of gesture recognizer, I created a new custom gesture recognizer named VerticalPanGestureRecognizer. This recognizer will perform the above work at the very beginning of a finger movement and register itself as a success or failure depending on whether or not delta Y was greater than delta X. The regular horizontal movement is captured by UIPanGestureRecognizer and this recognizer is conditioned to activate only if VerticalPanGestureRecognizer fails.

I am sure all of us have come across mobile applications that support multiple gestures on the same screen. It is easy to take for granted that these applications detect exactly what our intention is. With all of the above in mind, now we can have a little more appreciation for all such convenience!

ios 6 gesture recognizer 1 paul lee 1
Tech
Gilt Tech

New Gilt HQ

John Coghlan culture

The New Gilt HQ

Woohoo! Gilt and our colleagues at HBC Digital just moved into brand new offices! Our new digs are in Brookfield Place, a complex of office buildings across West Street from the Freedom Tower in Lower Manhattan. There’s a lot to love about the new space.

The office

It is great to have all of our NYC-based HBC and Gilt people in one location (we have teams in Dublin, St. Louis and Jackson and some remote engineers, too). We’re confident working side-by-side will allow us to better integrate our teams and better leverage our strengths.

Everyone is also really stoked about the views. The floorplan was designed to maximize the views and natural light by keeping the workstations and common areas on the perimeter while offices and conference rooms are in the center of the floor. Our location on the water provides great (I mean really great) views of the Statue of Liberty, New York Harbor, the Hudson River, Lower Manhattan and New Jersey.

The floor has dozens of conference rooms, a pantry, two large collaboration spaces and room for about 320 employees. Our ping pong and foosball tables will be arriving shortly.

The design of the space is clean and minimalist with lots of white and grey touches. Everyone is loving their new super comfy Herman Miller Aeron chairs and the cool little sofas in the collaboration spaces. We have some nice portraits on the walls already and our creative teams are working on a mural for one of the bigger walls to give the floor some more character.

The neighborhood

Brookfield Place is an awesome place to work. We’re right on the water and there is a marina with a sailing school and a handful of yachts right outside our doors. Our building is surrounded by grassy parks with volleyball courts, basketball courts and a skatepark in walking distance. The waterfront is car-free which gives the area a quiet and peaceful vibe.

When it comes to food, the variety is amazing. In our building alone we have the huge French marketplace Le District and Hudson Eats, a food hall with a bunch of NYC hot spots like Chop’t, Mighty Quinn’s, Num Pang, Umami Burger, Dos Toros and Blue Ribbon to name a few. Around the neighborhood, we have Shack Shack, Parm, a bunch of great coffee shops, multiple Starbucks (of course) and Eataly, which just opened across the street.

Other area amenities include an Equinox gym, tons of shopping, waterfront bars, Citibike stations, bike parking in our building and a ton of transit options.

The welcome

Of course, a new office needs to get broken in and People Operations was on it. We had breakfasts, lunch, a champagne toast and a watermelon social to help keep the excitement level high and give everyone a chance to meet their new neighbors since this was the first time our Gilt and HBC teams would be fully integrated on the same floor. The events were a hit and everyone seems be settling in nicely. Mission accomplished.

The invite

Want to see the new space for yourself? Stay tuned to our blog for info about upcoming tech events we’ll be hosting in the new space.

office space 1 culture 29 john coghlan 1
Tech
Gilt Tech

New open source project: scala-fedex

Ryan Caloras open source

We recently made the decision to switch from Newgistics to FedEx SmartPost for customer returns at Gilt. A couple of factors contributed to the decision, but the prettiness of FedEx’s API was not one of them - it’s not exactly the most developer friendly API you can find.

FedEx Web Services are a collection of APIs to do everything from generating shipping labels to tracking packages. Unfortunately, they’re still using SOAP and WSDLs! That means poor support for a modern language like Scala. Their sample Java client contained a bunch of unfriendly Scala code (e.g. Xml, blocking requests, and Java Classes rather than native Scala).

Enter scala-fedex

Using scalaxb we were able to generate a native non-blocking Scala client from FedEx’s WSDL. We then added a wrapper to further reduce the boilerplate of the raw generated client. The final result being a thin async Scala client for FedEx’s Ship Service API. It also provides a much cleaner example of usage than the previously mentioned Java code. We can now use this to generate FedEx return labels on Gilt using native reactive Scala code!

To support the community, we decided to open source and publish scala-fedex. You can find more specifics on the scala-fedex repo.

scala 13 fedex 1 open source 61 web-services 1 Ryan Caloras 7 John Coghlan 1
Tech
Gilt Tech

Rookie Summer

Team Rookie internship

The Summer Internship #TeamRookie

What can four interns do in a ten week internship? A lot, as it turns out.

With some help from the fantastic team led by Kyle Dorman, we built out a complete backend and frontend iOS app and had a ton of fun in the meanwhile. Here’s a snapshot of our summer experience.

Our internship began with two weeks of heavy learning. We were going to be building a mobile application using Swift, Scala, and the Play framework, but most of us had no prior experience with any of these tools. So our mentors led us through lab assignments to get our feet wet. We built a few simple iOS apps and a small Play application that used the Gilt Public API. As we became familiar with the languages and tools we would eventually use to build our intern project.

After our brief introduction to software development at Gilt, we started on a project of our own. Our mentors introduced us to Agile sprints, and we were off. Our progress was slow at first, but kicked up near the end, and on the last day of development we finally got our finished app deployed.

While we worked a lot this summer, Gilt made sure that we had tons of fun as well. Every Friday, we left the office early to explore a new part of New York. We went up the Freedom Tower, walked the High Line, and saw a musical on Broadway, amongst tons of other fun activities. We even had fun inside the office. We had weekly lunches with different teams across the organization, and we enjoyed random festivities like Cheese-mageddon, where tech samples more cheese than is probably healthy.

So, what was our actual project for the summer? The Gilt Style Quiz is a fun, playful way for users to start interacting with the Gilt iOS app without making a purchase. At the same time, the app gives Gilt a chance to get to know our users better as well. Through a series of style related questions, we are able to collect both brand and product category affinities for a user and are able to, as a result, better personalize the shopping experience for our users. As a team, we took complete ownership of the project and built the app from ground up. We began by developing the API and data models, and then we split up to tackle the front and back ends of the project.

What about Gilt Tech made the internship so cool?

Micro service architecture

Gilt uses micro services architecture to back the business operations. Because our service could be small and totally independent, we were able to make all of the design decisions in architecting the backend, super cool as an intern! We created a RESTful API with apidoc, an opensource project for declaring the models and resources of a RESTful API that comes with a suite of client generators.

#### Mentorship

Gilt provided a lot of resources to help us succeed and grow this summer. Right from the start we were introduced to individual mentors who helped us every step of the way, from learning Scala, Swift, and the magic that is AWS, to polishing out our product in the final week. Throughout the summer we had the opportunity to dine with the various tech teams and learn about the architecture supporting the Gilt backend and frontend. Erica also organized for us lunches with several executives from within Gilt and HBC, giving us firsthand insight to what drives the company.

From a project perspective, we had the chance to work with an amazing product manager, Hilah Almog, who defined our metrics of success and the scope of the application, as well as with a designer, Jose Salamone. It is easy sometimes to get caught in a cycle of code, test, and deploy without stopping to think of who is going to be using the product. However, getting the chance to work with non-engineers really helped keep the project in perspective. We weren’t writing code just to develop our skills in Scala and Swift or even to gather data for the personalization team. Primarily, we were developing a product to enhance customer satisfaction, to show our users that shopping on Gilt is a fun, enjoyable experience, and to streamline their transition into using their personalized product feed. While developing our technical skills was important, one of the key takeaways from this summer was definitely that it is crucial to keep your users in mind while developing!

Culture

Gilt has a unique, forward-thinking culture. The company constantly evaluates the tools it uses, and it is always open to exploring new technologies. We were exposed to this at the quarterly architecture council, where all the engineers spend a day discussing the current state of the Gilt technology stack and exploring possible new directions for tech.

Gilt is also committed to open-source and we made use of some Gilt open-source technologies in our project. The Cleanroom Initiative is an open-sourced codebase in Swift providing help with data transactions in our application. We also used an open-source apidoc Swift client generator and worked with the owner to make some additions to the project.

Takeaways

Throughout the summer we were exposed to a host of skills that eased the software development process. The adoption of standardized git workflows and explicit communication on project status through Agile sprints accelerated the development of our project across the short time frame. If we listed all we learned this summer it would take about, oh say, ten weeks, but needless to say this was a summer that we’ll all remember for a long time.

internship 3 mobile 22 team rookie 1
Tech
Gilt Tech

How to convert fully connected layers into equivalent convolutional ones

Pau Carré Cardona deep learning

The Problem

Say we want to build system to detect dresses in images using a deep convolutional network. What we have is a database of 64x128 pixels images that either fully contain a dress or another object (a tree, the sky, a building, a car…). With that data we train a deep convolutional network and we end up successfully with a high accuracy rate in the test set.

The problem comes when trying to detect dresses on arbitrarily large images. As images from cameras are usually far larger than 64x128 pixels, the output of the last convolutional layer will also be larger. Thus, the fully connected layer won’t be able to use it as the dimensions will be incompatible. This happens because a fully connected layer is a matrix multiplication and it’s not possible to multiply a matrix with vectors or matrices of arbitrary sizes.

Let’s assume we have 1024x512 pixels images taken from a camera. In order to detect dresses in an image, we would need to first forward it throughout the convolutional layers. This will work as convolutional layers can adapt to larger input sizes. Assuming the convolutional and max pool layers reduce the input dimensions by a factor of 32, we would get an output of 32x16 units in the last convolutional layer. On the other hand, for the training and test images of 64x128 pixels, we would get an output of 2x4 units. That size of 2x4 units is the only one the fully connected layer matrix is compatible with.

Now the question is, how do we convert our camera 32x16 units into the fully connected 2x4 units ?

The Wrong Solution

One way to do it is by simply generating all possible 2x4 crops from the 32x16 units. That means we would generate 403 samples of 2x4 units ( (32 - 2 + 1) x (16 - 4 + 1) = 403 ). Finally, we would go one by one forwarding those 403 samples throughout the fully connected layers and arrange them spatially.

The problem with that approach is that the cost of cropping and forwarding images throughout the fully connected layers can be impractical. On top of that, if the network reduction factor is lower or the camera images have a higher resolution, the number of samples will grow in a multiplicative way.

The Right Solution

Fortunately, there is a way to convert a fully connected layer into a convolutional layer. First off, we will have to define a topology for our fully connected layers and then convert one by one each fully connected layer. Let’s say we have a first fully connected layer with 4 units and a final single binary unit that outputs the probability of the image being a dress. This diagram describes the topology:

Converting the first fully connected layer

The idea here is to transform the matrix A into a convolutional layer. Doing that it’s pretty straightforward as the rows of the matrix A can be interpreted as convolutions applied to the flattened input V.

Let’s first write down the classical deep learning convolution operator:

When both the signal and the filter are of the same size, the convolution will generate a vector of size one. Hence, the convolution will be equivalent to the dot product:

Applying this property to our convolutional conversion task, we will be able to transform a linear operator into a vector of convolutions:

Therefore, we have the following transformed convolutional layer for the first fully connected layer:

More formally, we will have as many feature maps as rows the matrix A has. Furthermore, the i-th feature map will have as filter the i-th row of the matrix A.

Here we are assuming that the input of the fully connected layer is flattened and also that the fully connected layer only receives a single feature map from the last convolutional layer. For multidimensional convolutions with many feature maps, the transformation will depend on the way the framework we use encodes the different layer types (convolutional and fully connected).

In case of Torch, it’s pretty easy as one simply has to copy the biases and the weights of the fully connected layer into the convolutional layer. The caveat is that the convolutional layer has to be declared using the following parameters:

  • Number of input feature maps: as many as output feature maps the last convolutional layer has.

  • Number of output feature maps: number of outputs the fully connected layer has.

  • Filter dimensions: the dimensions of the output of each feature map in the last convolutional layer (we assume the all of the feature maps have the same output dimensions).

Converting the second fully connected layer

After the first transformation we will have in the second fully connected layer an input that has many feature maps of size one.

The equivalent convolutional layer will be the following:

  • Number of input feature maps: as many input feature maps as output feature maps the last transformed convolutional layer has. It will also be equivalent to the input units the original second fully connected layer has.

  • Number of output feature maps: as many output feature maps as outputs the second fully connected layer has. In our case we have a single output and therefore the layer will only have a single output feature map. In case we would have more outputs or an additional fully connected layer, we would need to add more feature maps.

  • Filter values: the filter architecture is pretty simple as all the input feature maps have units of size one. This implies that the filters will be of size one. The value of the filter in the feature map that connects the n-th input unit with the m-th output unit will be equal to the element in the n-th column and the m-th row of the matrix B. For our specific case there is one single output, thus m is equal to 1. This makes the transformation even easier. Nevertheless, we should keep in mind that we could potentially have multiple outputs.

For our example, the second fully connected layer will be converted into the following convolutional layer:

Always be convolutional

In this post we’ve discussed how to transform fully connected layers into an equivalent convolutional layer. Once the network no longer has fully connected layers, we will be able to get rid of all the problems they cause when dealing with inputs of arbitrary sizes.

Nevertheless, when designing a new neural network from scratch it’s always a good idea to design it substituting all fully connected layers with convolutional layers. This way, there is not only no need for any conversion but we will also get far more flexibility in our network architecture.

deep learning 1 convolutional neural networks 1 image detection 1 torch 1 Pau Carré Cardona 1
Tech
Gilt Tech

Akka HTTP Talk with Cake Solutions

meetups

We are thrilled to be hosting Aleksandr Ivanov of Cake Solutions on Tuesday, April 12th. He’ll be presenting an excellent talk on Akka HTTP. Who is Aleksandr? We’re glad you asked:

Aleksandr Ivanov is a senior software engineer at Cake Solutions, one of the leading European companies building Reactive Software with Scala. Scala is his main language of choice since 2011. He’s taken part in various projects, from developing backends for trivial web applications to enterprise-level reactive system and machine learning apps.

Beside engineering, he’s taking an active part in the life of the developer community, giving talks at local meetups, writing articles and helping others on the mailing list, Gitter and Stack Overflow. He’s always ready to hear and discuss interesting projects and events, share his experience or simply have a nice conversation over a drink.

Refreshments will be served!

Important: You must provide First Name & Last Name in order to enter the building.

Please RSVP!

meetup 8 cake solutions 1 akka 3 akka http 1 scala 13
Tech
Gilt Tech

Urgency vs. Panic

Hilah Almog tech blog

Urgency vs. Panic

My first initiative as a product manager at Gilt was something called “Urgency”. It was formed under the premise that Gilt customers had become numb to the flash model, and we in tech could find ways to reinvigorate the sense of urgency that once existed while shopping at Gilt; the noon rush wherein products were flying off the virtual shelves, and customers knew if they liked something they had precious few minutes to purchase it before it’d be gone forever. I came into this initiative not only new to Gilt but also new to e-commerce, and I felt an acute sensitivity towards the customer.At Gilt we walk a fine line between creating urgency and inciting panic, and it’s something I personally grappled with continuously. The former’s outcome is positive, the shopping experience becomes gamified, and the customer’s win is also ours. The latter’s outcome is negative. The customer has a stressful and unsuccessful shopping experience, and then churns. This fine line meant that we as a team couldn’t just conceive of features, we also had to find the perfect logical balance as to when they should appear – and more importantly, when they shouldn’t.

Cart Reservation Time

Our first feature was reducing the customer’s reservation time by half once they add a product to their cart. This tested well, but felt mean. We therefore held its release until we could build a product marketing campaign around it that communicated the shorter time as an effort in fairness: “if other customers can’t hoard without real intention to buy, then you get the most coveted products faster”. The customer service calls ended once our shoppers felt the feature was for their protection, not harm.

Live Inventory Badging

We wanted to continue running with this theme of helpful urgency, leading us to our second feature: live inventory badges. When we have less than 3 of any given item, a gold badge appears atop the product image saying “Only 3 Left”. It then animates in real time as inventory of that item changes. If you are ever on Gilt right at noon, notice how our sales come alive through these badges. Unlike the cart reservation time, this feature felt like a one-two punch. Not only were we creating urgency, but we were also giving the customer something they rarely get while shopping online – a view of the store shelf.

Timer in Nav + Alerts

Our third feature was our biggest challenge with regard to striking the right balance between urgency and panic. We added a persistent cart timer in the navigation, showing how much of your aforementioned five-minute reservation had transpired. The timer’s partner in crime is an alert, in the form of a banner, that appears on the bottom of the page when only a minute is left on your item’s reservation, urging you to checkout before it’s gone.

In order to find ourselves on the right side of the line, we implemented stringent rules around when this banner could appear, limiting it only to products that are low inventory (less than 3 in your size), and once per session.

Live Views

We faced an altogether different challenge when it came to our final feature, live product views. Here, the feature itself wasn’t strong enough, the views had to carry their weight. We again were forced to think through very specific thresholds depending on inventory levels and view count in order to determine under what circumstances we show, and under which we hide the feature.

Each of these features were tested individually, and each yielded positive results. After each was released we saw a combined 4% increase in our Key Performance Indicators on revenue within the first hour of a sale. The line was traversed successfully without panic but with the intended effect. And to our customers we say; Because you’re mine, I walk the line.

KPI 1 urgency 1 panic 1 shopping 14
Tech
Gilt Tech

Breaking the Mold: Megaservice Architecture at Gilt

Adrian Trenaman aws

Today we announce a novel approach to software and system architecture that we’ve been experimenting with for the last while at Gilt: internally, we’ve been referring to it ‘mega-service’ architecture, and, the name seems to have stuck. We’re pretty excited about it, as it represents a real paradigm shift for us.

In a mega-service architecture, you take all your code and you put it in one single software repository, the mega-service. There are so many advantages to having a single repository: only one code-base; no confusion where anything is; you make a change - it’s done, and will go out with the next deploy. It all compiles, from source, 100% of the time at least 50% of the time. Software ownership is a perpetual challenge for any tech organisation: in the mega-service model, there are many, many owners which means of course that the code is really, really well owned.

The mega-service is deployed to one really big machine: we prefer to run this in our own ‘data centre’ as we believe we can provision and run our hardware more reliably and cost-effectively than existing cloud players. The benefits of having a mega-service application are manifold: there’s one way to do everything and it’s all automated; code instrumentation, configs and metrics are all consistently applied, and, all eyes are on the same project, scripts and code, so people are more familiar with more parts of the system.

We’ve abandoned the sophisticated distributed code control mechanisms of recent lore in favour of a big ‘directory’ hosted on a shared ‘file server’. We’ve resorted to an optimistic, non-locking, non-blocking, zero-merge, high-conflict algorithm called ‘hope’ for contributing code changes: we copy the changes into the directory, and then ‘hope’ that it works. Rather than work with multiple different programming languages and paradigms, we’ve settled on an ‘imperative’ programming style using a framework we’ve recently adopted called Dot Net. Aligning previous lambda-based actor-thinking to a world of mutable variables, for-loops and ‘threads’ has not been easy for us; however, we suspect that the challenges and difficulties we’re experiencing are mere birthing pains and a clear sign that we’re heading in the right direction: if it’s hard, then we must be onto something.

This new architectural approach is an optimization on Neward’s ‘Box-Arrow-Box-Arrow-Cylinder’ pattern, reduced to a profoundly simple ‘Box -Arrow-Cylinder’ diagram (despite forming an elegant visual, the solution is just slightly too large to fit in the margin). We typically draw a box (our monolithic code) on top of a cylinder (our monolithic database), both connected with a line of some fashion; however, some have drawn the box to the left, right or bottom of the cylinder depending on cultural preference. Distinguished Engineers at Gilt have postulated a further simplification towards a single ‘lozenge’ architecture incorporating both code and data store in a single lozenge: while that architecture is theoretically possible, current thinking is that it is unlikely that we will get to prototype this within the next ten years.

New architectures require new thinking about organisational structure: everything so far points to a need for a software organisation of about five Dunbars in size to maintain our code-base, structured with a golden-ratio proportion of about eight non-engineering staff to every five engineers. Additionally, the benefits of really thinking about and formalizing requirements, following through with formal design, code and test over long periods, in an style we refer to as ‘Radical Waterfall’, bring us to a rapid release cycle of one or two releases per solar year.

While most readers will be familiar with open-source contributions from gilt on http://github.com/gilt and our regular talks and meetups, the innovations described in this post are subject to patent, and available through a proprietary licence and submission of non disclosure agreement. We’ll be releasing details of same on our next blog post, due for publication a year from now on April 1st, 2017.

aws 5 codedeploy 2 newrelic 2 notifications 2 micro-services 22 april-fool 1
Tech
Gilt Tech

Front End Engineering Lightning Talks with HBC Digital

meetups

Join us for an evening of lightning talks by 4 of HBC Digital’s Front End Engineers and an introduction by Steve Jacobs, SVP, Digital Technology and Demand Marketing.

  • Ozgur Uksal - Front End Engineer: Working with Typescript
  • Lei Zhu - Front End Engineer: Redux
  • Norman Chou - Front End Engineer: Using React router
  • Rinat Ussenov - Front End Engineer: Juggling bits in Javascript

Refreshments will be served!

Important: You must provide First Name & Last Name in order to enter the building.

Please RSVP!

meetup 8 hbc digital 1 frontend 12 typescript 1 redux 1 react 1 javascript 11
Tech
Gilt Tech

OSX, Docker, NFS and packet filter firewall

Andrey Kartashov docker

The Mobile Services team at Gilt uses Docker to both build and run software. In addition to the usual Docker benefits for software deployments moving toolchains to Docker has a few advantages:

  • it’s easy to (re)create a development environment
  • the environment is preserved in a stable binary form (libs, configs, CLI tools, etc, etc don’t bit rot as main OS packages or OS itself evolve)
  • easy to support multiple divergent environments where different versions of tools/libs are the default; e.g. java7/8, python, ruby, scala, etc

We develop primarily on OSX, but since Docker is a Linux-specific tool, we must use docker-machine and VirtualBox to actually run it. Toolchains rely on having access to the host OS’s filesystem. By default /Users is exposed in the Docker VM. Unfortunately, the default setup uses VBOXFS which is very slow. This can be really painful when building larger projects or relying on build steps that require a lot of IO, such as the sbt-assembly plugin.

Here’s a great comparison of IO performance.

There’s really no good solution for this problem at the moment, but some folks have come up with a reasonable hack: use NFS.

One of them was even nice enough to wrap it up into a shell script that “just works”.

Indeed, with NFS enabled, project build times begin to approach “native” speeds, so it’s tempting. The issue with NFS continues to be its aging design and intent to function in trusted network environment where access is given to hosts, not to authenticated users. While this is a reasonable access model for secure production networks, it’s hard to guarantee anything about random networks you may have to connect to with your laptop, and having /Users exposed via NFS on un-trusted networks is a scary prospect.

OSX has not one but two built-in firewalls. There’s a simplified app-centric firewall available from Preferences panel. Unfortunately all it can do is either block all NFS traffic (docker VM can’t access your exported file system) or open up NFS traffic on all interfaces (insecure), so it doesn’t really work for this case.

Fortunately, under the hood there’s also a much more flexible built-in packet level firewall that can be configured. It’s called PF (packet filter) and its main CLI tool is pfctl. Here’s a nice intro.

With that, one possible solution is to disable firewall in the Preferences panel and add this section at the end of the /etc/pf.conf file instead:

# Do not filter anything on private interfaces
set skip on lo0
set skip on vboxnet0
set skip on vboxnet1

# Allow all traffic between host and docker VM
table <docker> const { 192.168.99/24 }
docker_if = "{" bridge0  vboxnet0  vboxnet1 "}"
pass quick on $docker_if inet proto icmp from <docker> to <docker>
pass quick on $docker_if inet proto udp from <docker> to <docker> keep state
pass quick on $docker_if inet proto tcp from <docker> to <docker> keep state

# Allow icmp
pass in quick inet  proto icmp
pass in quick inet6 proto ipv6-icmp

# Bonjour
pass in quick proto udp from any to any port 5353

# DHCP Client
pass in quick proto udp from any to any port 68

# Block all incoming traffic by default
block drop in
pass out quick

Then turn it on at a system boot time by adding /Library/LaunchDaemons/com.yourcompany.pfctl.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Disabled</key>
	<false/>
	<key>Label</key>
	<string>com.gilt.pfctl</string>
	<key>WorkingDirectory</key>
	<string>/var/run</string>
	<key>Program</key>
	<string>/sbin/pfctl</string>
	<key>ProgramArguments</key>
	<array>
		<string>pfctl</string>
		<string>-E</string>
		<string>-f</string>
		<string>/etc/pf.conf</string>
	</array>
	<key>RunAtLoad</key>
	<true/>
</dict>
</plist>

And configuring it to start by running

sudo launchctl load -w /Library/LaunchDaemons/com.yourcompany.pfctl.plist

The main difference from /System/Library/LaunchDaemons/com.apple.pfctl.plist here is the addition of -E parameter.

You can check that it starts by default after a reboot with

sudo pfctl -s info

And check the rules with

sudo pfctl -s rules

It should be ‘Enabled’ and you should see the configured rules.

You can verify your overall setup by running nmap from a different node against your laptop, e.g.

sudo nmap -P0 -sU  YOUR_IP
sudo nmap -P0 -sT  YOUR_IP

And check for open ports.

After that configuration you should see a noticeable improvement in docker performance for file system intensive workloads. Hopefully this will no longer be necessary in future versions of Docker VM so check the docs to be sure.

docker 2 osx 1 nfs 1 firewall 1
Tech
Page 1 of 65