Foreman and Change Management+VCS with Puppet


Just so you know, the Systems Administration team here at Gilt is lazy.  We are constantly looking for ways to make our jobs easier, and automate everything possible.   When it comes to having to say, update a package on every server in our sizable network; we respond with, “That just sounds like a lot of work, I wonder if there is something that can do all that for us, so that we can keep reading web comics?”

Puppet Labs gave us the answer.

In case you didn’t know, Puppet is a pretty amazing tool that allows us to automate many of the hopelessly repetitive yet necessary tasks of systems administration.  It lets us keep all our systems configurations up to date, and uniform across our enterprise; and it lets us do it without typing our fingers off.

The Systems Administration team here at Gilt loves Puppet.  As it turns out, so do a ton of other systems administrators in the NYC area.  Which is why we give aid and comfort to the New York City Puppet Users Group.  This time around it was hosted in Gilt’s offices in Midtown.

This week featured two topics:

1. Foreman Node Classifier

Eric E Moore, from Brandorr Group LLC, gave a great demo and overview of Foreman.  Foreman is a web front end, external node classifier, and life-cycle management tool for puppet.  Eric shows us how Brandorr leverages Foreman to help with puppet configuration and administration.  He demonstrated how Foreman can be used to monitor puppet runs for errors, and track its runs across multiple servers.  It’s a great tool. 

We are considering using Foreman in house at Gilt.  Eric gave us some great tips.

Check out Foreman’s website: http://theforeman.org


2.    Change Management and VCS work flow with Puppet.

Presented by Garrett Honeycutt from Puppet Labs, this talk was focused on how to best use tools like SVN and GIT to keep all the teams that use a Puppet implementation together.   His view on it is that it’s less of a technical problem, and more a problem of change management.  He pointed out that having a good, consistent policy goes a long way here, and the positives of using test-driven development.  The slides from his talk are available here:

http://talks.garretthoneycutt.com/20110413-Change_management_with_Puppet_for_PuppetNYC.pdf


Both talks were outstanding.  Made even more outstanding by the all of us heading out for drinks afterward.   The NYC Puppet Users Group meets every month, and the Systems Administration team at Gilt loves being a part of it and sharing ideas, and beer, with our colleagues at other companies.

If you are interested in joining in on the fun, check out the NYC Puppet User Group here: http://groups.google.com/group/puppet-nyc

Slides and video from Eric Shepherd’s presentation given at Gilt HQ and at the Brooklyn JavaScript Meetup:

Summary

For modules to function within a large-scale system and on third-party sites, they need to be self-contained units with minimal dependencies. They also need to keep their hands off of other modules and library code. Gilt’s module framework manages multiple independent components, providing them with what they need, and only what they need, to do their jobs.

Once the module framework is built, third parties still need an easy way to consume modules. A server-generated JavaScript bootstrap file allows all the module’s options to be wrapped in a closure, keeping the DOM clean and conflict-free. There are some complex problems that arise when embedding code this way, so be sure to watch for them!

Building a JavaScript Module Framework at Gilt from Gilt Tech on Vimeo.

Building a JavaScript Module Framework at Gilt

View more presentations from Eric Shepherd

Gilt Groupe’s KIVA robots moving merchandise to and fro.

Templating is a powerful way of separating concerns on the front end. It’s a natural fit for Gilt’s platform, since much of our content comes to the front end from services that return JSON. Sometimes that JSON is rendered in JSPs with JSTL, but sometimes it’s better to render it with JavaScript.

Until recently, most JavaScript HTML rendering involved messy string concatenation, lots of looping, and an unholy marriage of logic and content in the same file. However, new templating engines, such as Handlebars, allow multiple logic-less templates to be created for the same content, with clean, easy-to-read syntax. Compare:

But which template engine to use? There are many currently vying for prominence. I didn’t want to hitch our wagon to any one engine, since many different pieces of Gilt code need to make templating calls and none of us knows the future, so I decided to create a middle tier API that would accept any templating engine as a plugin. This is generally a good way to approach front end architecture, as it’s all too easy to scatter third-party dependencies throughout a large codebase, leading to difficult maintenance and requiring significant restructuring if the third-party code ever needs to be changed.

What It Is

There are three components in addition to the implementation code:

• Gilt.Template, which exposes render(), register(), plugin.add() and plugin.get() methods
• a template file saved using a directory and file naming convention
• the chosen rendering engine code

How It Works

1. Templates are stored in a Gilt.Template private cache, by a unique combination of name, engine, and version. Name is typically the type of feed or data that is being rendered, such as a wait list, a single product feed, or a cart. Version indicates which view of the data is required (“default” is the canonical view, and views can exist for third-party sites or other areas on the Gilt site such as a modal window or a specific page). Engine indicates which third-party rendering to use, with Handlebars as the default if none is specified.
2. For flexibility, all rendering is asynchronous. The render() method does not actually write any markup; instead, it supplies the rendered markup to a passed callback function. This allows the template file or rendering engine to be lazy loaded at the time of invocation, rather than be already present on the page, and it recognizes that often the rendered template is needed for purposes other than writing immediately into an element on a page.
3. Each template engine is an equal citizen, registered as a plugin with Gilt.Template. Handlebars and Dust are built-in, as we use both of them on the Gilt site; however, any additional template engine can be registered and used. Each plugin must provide a render() method, and optionally can provide a register() method (used, for example, in Dust compilation). When the implementation code calls Gilt.Template.render(), that method delegates to the plugin’s render() method.
4. If the template is not available in Gilt.Template’s cache, it triggers an asynchronous load of the template file, looking by a naming convention of /templates/<type>/<version>.<engine>.js. For example, a template that handles the default view of a wait list feed using Handlebars would be stored at /templates/wait_list/default.handlebars.js. This convention allows for multiple views of the same data feed. Gilt.Template.render() takes an optional argument for an ajax service to use, with jQuery used by default. This allows for the possibility of use with a different base library, since this is the only point of third-party dependency.

How To Use

So, the three steps required to use the system are:

1. Add a plugin using Gilt.Template.plugin.add() with the specific code required to interact with the chosen third-party templating engine. Handlebars support is built-in and, for reference, looks like this:

2. Register a template with Gilt.Template.register(), indicating a unique combination of name, engine, and version. This could be done prior to calling render(); if not, render() will lazy load this template file, thereby calling register():

3. Call Gilt.Template.render(), passing it the unique name, engine, and version to render, along with a callback to be provided with HTML. The render() method will load the plugin and call register() first if the template is unavailable. The following code is probably in the callback of an ajax request function:

tl;dr;

In summary, Gilt.Template:

• allows for template storage by engine, version and name;
• provides flexibility by providing rendered markup to a callback function;
• treats all templating engines equally; and
• speeds up page load by asynchronously loading template files.

Templates have helped us to separate our content data from its views on the site, and our middle tier architecture has provided a consistent API and eliminated the need for third-party calls in the Gilt codebase.

While working on TestSwarm, we ran into the problem that none of the javascript testing frameworks out there had really solved the problem of how to test asynchronous behavior without requiring the test writer to do something different from how (s)he would test non-asynchronous behavior.

e.g. Testing these two lines should be done the same way:

callThisImmediately(function () {});
callThisIn5Seconds(function() {});

Every other testing framework we looked at required using something like wait() and resume(), or mock timers, or completely ignored the problem.

So… I decided to write my own testing framework, with testing asynchronous behaviors (like ajax requests, animations, countdowns, timeouts, etc) in mind from the very beginning. Since we also have a lot of Rubyists here, we wanted the syntax to be as close to RSpec as possible.

Thus was born ASpec:

For the most part this was fairly straightforward, but there were two itty bitty little problems. The first was of course… how does ASpec know that there is asynchronous behavior? And two, what should it do about it?

We solved the first one using a little regex and Function.prototype.toString(). Simply put, ASpec counts the number of expects in an it block before it executes that it block. when an expect gets executed, we count that too. When we exit the it block, if the two numbers aren’t the same, well, chances are that there was asynchronous behavior. Either that or unreachable code, conditional statements, or a good old runtime error.

As for the 2nd problem, we solved that too. If we have any pending expects, everything after that gets put into a queue, and is executed after the pending expect is done. Or times out.

When everything is done, we wind up with a nice little report that TestSwarm can take a screenshot of:

Well, that’s okay (sorry about the purple), but if I’m testing something, I’d rather not have to leave the command line in order to do so. Not to mention, I’d like to have a way to test all my specs at once. And of course, I’d like to be able to test everything before I commit.

Therefore, we created a little rake task: rake spec. If you’re a Rubyist, it probably sounds familiar.

Our version of rake spec even produces output similar to the RSpec version:

If you peek under the hood, what you’ll see is a couple simple minded tcp servers, and the Mac OSX open command (with the --background flag).

And of course, since we want to be able to test in different browsers, we can run rake spec:firefox, rake spec:chrome and rake spec:safari. Sorry, IE.

Kevan Davis Gilt Groupe

When you’re developing an iPhone or iPad application, running the app on the device is essential to understanding whether your code is working correctly. Although the iOS Simulator is very useful for testing functionality and user interface interactions, it doesn’t provide a very good simulation of the memory constraints your app will face on actual devices. As a result, applications that run fine in the simulator may be unexpectedly terminated by the operating system when run on devices.

Often, but not always, crashes that happen only on the device are due to memory warnings that are not being handled correctly by your application. Unfortunately, if your application terminates unexpectedly on the device, you’re not always given a clear indication of the cause.

One tool for diagnosing such problems is the Xcode Organizer. The left-hand sidebar of the Organizer window contains a list of the various devices that have been hooked up to your computer over time. While a development-enabled device is connected to your computer, the Organizer window will show a green dot next to the device name, and the if you select a connected device in the “Devices” list, you will see a number of tabs at the top of the main part of the window.

The “Device Logs” tab should show any crash logs recoded on the device for the applications you’re developing. On my installation of Xcode, however, crash logs don’t always appear there, making it difficult to determine with any surety the cause of a given crash. (I’m not sure why certain logs are omitted from the Organizer; I just know that in my experience, they don’t always show up. I’m currently running Xcode 3.2.6.)

Fortunately, there is a work-around. If you sync a device through iTunes, it turns out that new crash logs are copied from the device to a location within your home directory. You can find the crash logs for your device(s) within:

~/Library/Logs/CrashReporter/MobileDevice

Inside that directory, you’ll see a directory for each device that (1) has been synced through iTunes, and (2) had crash logs to copy at the time it was synced. (The name of the directory will be taken from the name of the device.)

If a given application crashed due to an exception such as EXC_BAD_ACCESS, you should see a log file containing the name of the application’s executable followed by a timestamp indicating the time of the crash. The contents of the log file will give you more information about the nature of the problem.

When the operating system terminates applications due to low memory conditions, it will create a log file starting with the name “LowMemory-” followed by a timestamp. The file will list any application(s) terminated and give clues about the memory usage of each.

Even when running an application on the device within the debugger, console output isn’t always clear about the reason that an app was terminated. In those cases, the Xcode Organizer seems like the right place to look, but you might not find all the expected logs there. Syncing your device with iTunes and digging into the log files copied to your local filesystem is sometimes the only way to get the clues you need to diagnose certain types of unexpected terminations.

Evan Coyne Maloney
Senior Software Engineer, Mobile Team
Gilt Groupe

We’ve made a number of optimizations to our iPhone/iPad codebase over the past couple of months, the effects of which can be seen in the attached video:

(Because our application is so dependent on the network, the new code won’t appear to outperform the old code 100% of the time. This is due to the vagaries of the connection between the devices and the server.)

As we prepare to submit this latest code to Apple, we decided to make some of our optimization techniques public in the hopes that it may benefit developers outside of our company:

• In a number of places in our code, we were spawning new threads to perform short-lived operations. We’ve rewritten a number of these as NSOperations, allowing us to use NSOperationQueue instances, which are implemented under the hood by the OS as thread pools. So we’re spawning fewer new threads and reusing existing threads more; as a result, we’ve avoided some overhead of launching new threads. Also, since the OS is able to manage threading behavior for us via the NSOperationQueue, the OS is able to make smart decisions at runtime about thread resource usage that we were previously hard-coding at development time. Our assumptions could never be correct for all cases, since each device may have a different number of other processes running at any time. The OS itself is in the best position to make such decisions.
• Using the @synchronized construct for thread safety in iOS is very expensive. (Part of the reason for this is that @synchronized sets up inherent exception handlers.) So, one major optimization was to replace @synchronized blocks with using the NSLock/NSRecursiveLock classes for thread safety. Doing this reduced thread waits that occur during lock contention, improving app performance overall.
• Another way we avoid unnecessary locking is by using the thread-local storage provided by the threadDictionary: method of NSThread. Thread-unsafe resources that are expensive to construct—but that don’t take up a lot of memory—are good candidates for this. So, instead of constructing many short-lived objects on an as-needed basis, we can construct one per thread and keep it in thread-local storage. This ensures that the resources are used in a thread-safe manner, while avoiding the expense of repeated instantiations. (This technique is less useful for short-lived threads that are spawned often, because in those cases you won’t be avoiding the instantiation costs. Note, too, that Apple recommends against using thread-local storage for threads managed by NSOperationQueues.)
• We discovered that instantiating and using the NSDateFormatter class is particularly expensive. (And, because the NSDateFormatter class is not thread-safe, we can’t share instances across threads.) We use this class to parse text dates that arrive from the server. So, to avoid the instantiation expense, we’re caching NSDateFormatters in thread-local storage. Also, in our particular case, because so many of the date strings we’re parsing turn out to be identical (a result of many sales starting at the same time), we’re avoiding some of the date parsing altogether by maintaining a cache with date string keys that point to the NSDate instances that resulted from parsing that date string. Most of our dates now don’t need to be parsed at all; we just need to look up the NSDate associated with a given date string. Doing this shaved 5%-10% off the processing done at app startup.
• Our old caching system used a SQLite database; we’re now using the filesystem for caching, which avoids all the marshalling/unmarshalling required by the database accesses. This is particularly beneficial for our image cache. Now that cached images are stored in the filesystem, we can load them directly with [UIImage imageWithContentsOfFile:]. Previously, our image data was first loaded into an NSData instance that was then passed to [UIImage imageWithData:]. Not only does this new technique perform better, but by using this method to reference images, the OS has the option of purging image data from memory and reconstituting it directly from the file on an as-needed basis. (Whether or not a given version of iOS does this is another story, but at least this technique gives the OS that option.) That’s not possible if we were still using [UIImage imageWithData:]; the OS can’t purge the image data in that case since there would be no way to transparently reconstitute it if the image data were needed again. This gives the OS the ability to better manage memory on our behalf.
• Since a big factor in iOS app performance is the memory footprint of the application, we’re also making better decisions about what images to store in our memory cache. Large images in iOS are particularly expensive, since the operating system stores image data uncompressed in memory. The large Gilt sale images are over 6MB uncompressed, and we’re now avoiding caching those in memory altogether. (We now only cache images in memory if their uncompressed size is less than 1MB.) This reduces the amount of memory warnings our app has to handle, which means we don’t have to flush our memory caches as often. As a result, all of our various memory caches have more cache hits and fewer misses; our memory caches are now more efficient.

If there’s any interest in my going into further detail on any of these items, let me know. In the meantime, I hope you find this helpful!

Evan Coyne Maloney
Senior Software Engineer, Mobile Team
Gilt Groupe