I work as a lead software engineer for Gilt—my team focuses on our email operations, from design to distribution. I also spend a lot of time training for endurance events such as triathlons and long distance cycling. During the requisite hours and hours of preparing for these events, I spend a lot of time thinking about my workouts and my work at Gilt. One result of all this contemplating is that I’ve come up with an interesting parallel between how a successful athlete deals with stresses to the body, like excess lactic acid build-up—and how a successful business deals with the complexities of its business (ie excess systems build-up). Let me explain.

First Things First: What Is Lactic Acid? 

Lactic acid is a chemical that builds up in your muscles during intense exercise. Accumulation of this acid in your muscles leads to muscular fatigue. It’s that feeling you get after an intense run, when your muscles just can’t work anymore.

Your body removes lactic acid continuously through your blood stream.  A good way to understand this process is to visualize a cup with a hole in the bottom.  As you exercise, you are pouring water into the cup—causing more and more lactic acid to build up in your muscles.  Water drips out the bottom continuously, much like your blood removes the lactic acid from your muscles. As more lactic acid builds up, your ability to perform is reduced. It’s a non-linear scale. Increasing volumes of lactic acid reduce your ability to perform by greater and greater amounts.

What, Exactly, Constitutes Complexity in a Business?

My dictionary describes complexity as “many parts where those parts interact with each other in multiple ways.”  My own definition gives the term a pretty broad scope. I see “complexity” as being anything that changes the parts involved, or the interactions of those parts. A short list of things that count in my book:

  • Employee turnover
  • Hiring new people
  • Changing project focus
  • Employees changing roles
  • Etc…

As you might expect, I paint with a broad brush on the technical side as well:

  • New services
  • New build systems
  • New languages
  • New data centers
  • Different programming ideologies (i.e. reactive, dependency-injection, etc…)
  • New release procedures
  • Any changed lines of codes
  • Etc…

So, almost everything adds a bit of complexity. But the good news is that most complexity goes away with time. Bugs are crushed. Employees undergo onboarding and get up to speed. Old systems are decommissioned. Systems become more stable.

How Are Lactic Acid and Complexity Connected? 

An organization’s ability to handle complexity is very similar to the body’s ability to handle lactic acid. We can maximize throughput in an organization by monitoring the rate at which we allow complexity to seep in; this is very similar to what endurance athletes do every day. “Saving” a bit will allow their bodies to go further and faster. I think “saving” applies to businesses as well. Relating this to my work at Gilt: I pay a lot of attention to the rate at which we introduce complexity. I choose projects for my team that maintain a healthy amount of complexity in order to ensure high throughput.   

At Gilt, we introduced quite a bit of complexity by migrating between binary incompatible versions of Scala. Libraries needed to be built to support both versions. Projects needed to be upgraded. In the short- to medium-term, this made life hard. As time went on, however, much of that complexity disappeared as we got better at migrating projects and completed more of the overall migration. These natural reductions in complexity are similar to the blood removing lactic acid from muscles. 

Much like having too much lactic acid in your blood, introducing too much complexity at one time will slow down progress considerably. A business who wants to succeed (“win the race”) has to control the amount of complexities it introduces initially, and wait for its “blood stream” to cleanse some of the complexities before introducing new ones.

image

The first meetup at Gilt’s new office of the Dublin Scala Users Group  featured two great talks, a lively crowd of engineers, and our brand-new, brightly colored ComfyChairs (not trademarked)! For all of you who couldn’t make it, here are the slides from Citi Lead Mobile Architect Aman Kohli and Citi Software Engineer Kevin Yu Wei Xia’s talk, “Happy Performance Testing—DSLing Your System with Gatling”:

And here’s Gilt Senior Software Val Dumitrescu leading our audience into the CAVE:

image

CAVE is Gilt’s open-source, managed service for monitoring infrastructure, platform, and application metrics that provide visibility into your system’s performance and operational levels. Check out Val and Pawel Raszewski’s slides are here:

We were super-excited to be hosting a meetup in our own space for the very first time in Dublin! Stay tuned for the next installment. And if you’ve got a Scala talk to propose, please email your idea to lapple at gilt dot com!

Gilt’s NYC office hosts our first tech meetup of autumn 2014! On Wednesday, Sept. 24, Henrik Engström of Typesafe will present “What Constitutes a Reactive Application?”—a presentation on Typesafe Activator and the four traits that constitute a reactive application. Henrik will demonstrate, in code, how we use an reactive approach when building Typesafe Activator with Play and Akka. The emphasis will be on how Play is utilized, but there will also be some simple Akka examples. All code examples will be in Scala.

Go here to RSVP. Refreshments will be provided by Peroni and by Krave Jerky!

image

The Gilt team is excited to host the next Dublin Scala Users Group event at our awesome new office in Dublin 4! Here’s what is on the menu:

Happy Performance Testing—DSLing Your System with Gatling: In this talk, Citi Lead Mobile Architect Aman Kohli and Citi Software Engineer Kevin Yu Wei Xia will present their experiences using Gatling, a load-testing framework written in Scala. The power of Gatling is the DSL it provides to allow writing meaningful and expressive tests. Aman and Kevin will provide an overview of the framework, a description of their development environment and goals, and present their test results.

CAVE: An Overview:  Gilt Senior Software Engineer Val Dumitrescu will provide an overview of CAVE: Gilt’s open-source, managed service for monitoring infrastructure, platform, and application metrics that provide visibility into your system’s performance and operational levels. CAVE is built with Scala, Play and Akka.

The meetup takes place on September 23 at 7 PM; go here to RSVP.

image

Here’s Gilt Principal Software Engineer/Crafter O’Beers/Seasoned Instructor Gregor Heine leading one of his world-famous Scala trainings for our newer hires. (First Scala training in our schmancy new office!) Joining him as co-pilot for the first time is Gilt Senior Software Engineer Gary Coady, an expert in his own right. This class was Gilt-only, but we’re working on future installments for the public. Check back here for announcements!

imageThe Atlassian Summit is probably the top conference worldwide for project managers, program managers and anyone else who spends a lot of time thinking about agile development—and Gilt will be there! Gilt Senior Director, Program Management Office Heather Fleming and Director of Program Management Justin Riservato will share their talk “Beyond the Crystal Ball—The Agile PMO" on the second day of the conference, which takes place September 9-11 in San Jose, CA. Heather and Justin take the stage at 11:15 AM PST in Room 2.

But that’s not all: Gilt Lead Software Engineer Yoni Goldberg will give his super-popular talk on “Scaling Microservices at Gilt" (seriously—he’s delivered it in at least four U.S. states this year!) on Thursday, Sept. 11 at 11:10 AM PST in Room 7.

image

If you’re a Business Intelligence Engineer, we’d like to hear about your favorite tools and methodologies. Email us at lapple at gilt dot com.

Hello, George! Tell us a bit about how you arrived at Gilt.

I got here in 2012 after working in business intelligence at Everyday Health, MTV/Viacom, and Hewlett-Packard, and before that I went to school at Cornell.

How have things changed since the early days?

When I first joined the team, we were still trying to figure things out. Our data warehouse was new, our self-service framework wasn’t built out yet, and our goals weren’t as well defined. Since then we’ve streamlined our processes and become a lot more efficient, within the team and out. We were doing a lot of repetitive work and now there’s much more time to work on projects that will make a difference and that we’re passionate about.

What drew you to Gilt in the first place?

Just walking into the office I felt a buzz and got the sense that there’s a healthy atmosphere here. Gilt has a great reputation, with a lot of smart people working here. Everyone I spoke to during the interview process, I had a good connection with. Technology-wise, we’re pretty cutting-edge and I was excited to work with the team and learn new skills with the latest tools.

What does an average day look like for you?

Every day is different, but typically I spend half the day doing analyses, then working on projects for different teams. For example, I’m working with our customer transactions team to get credit card data into our data warehouse, and our loyalty team on getting new AB testing data into the warehouse as well. The data team will help the engineers figure out how the data model will work and look. Projects vary in length and scope, but the ones I’m working on now have lasted a couple months.

The data team’s recent work has been involving a lot of open source technologies.

Yes. We’re doing much of our current work through new event streams, in which services relay messages to Kafka, and the data warehouse reads those messages. Eventually we’ll use Hadoop for the file storage, with Kafka relaying the messages there and the data warehouse reading from Hadoop. In our work with Kafka, the idea behind it—and we’re not there yet—but it is to create easy access to all of the data within the company, so no one will have to ask an engineer to create a process to access a file or send data to the data team; they can just publish events to Kafka.

In terms of technology skills, what should a business intelligence engineer possess?

Prior experience using a range of business intelligence tools, a solid background in SQL, and an understanding of how a data warehouse works. But the primary qualification for the job is a good mind for analytics—the ability to draw connections between different data sources, understand data models, see patterns, and consolidate them into insights decipherable by anyone.

At Gilt, a lot of our data isn’t exact, so working with it involves a lot of research. At a more granular level, that means working with various engineering teams to understand the data they’re producing and putting the data together from multiple sources (inventory, site-live inventory, email, click-stream, order transactions) to fit it into a model. Then we run the data through the model to understand its various attributes, and highlight the important attributes to support the various business decisions we make.

One of the things we pride ourselves on here is our flat structure. Do you get to work with tech leadership much?

I do work with the tech execs occasionally, but I work more frequently with the heads of other departments, like Marketing and Merchandise Planning. The data team works with every department in the company outside of tech: marketing, finance, operations, merch planning, etc. The projects we work on relate to everything from supply chain management, to new marketing initiatives, to understanding our customers and demographics, to AB testing and understanding more about the user experience. Across these initiatives, my role is to understand where all the data comes from, put it all together and make it easy to use for everyone else in the company who needs it. Within the data team, I work closely with our chief data scientist to run data models and see if certain assumptions about our data make sense or not.

Talk a bit about the tech stack our data team uses.

Our stack currently includes Aster Teradata, Hadoop, Kafka, AWS and AWS Kinesis, real-time data events streams, and lots of micro-services written in Scala. On the front-end we work with Cognos, Looker and Spotfire.

What do you enjoy most about working with this stack?

Our data warehouse is pretty amazing in terms of speed and processing power. The amount of data we can process, run analytics on and get responses back—and quickly—is pretty awesome. A lot of other places I’ve been to, especially startups, don’t have the infrastructure to do that. It’s great to be able to iterate on your analytics quickly—you don’t have to wait for the data to come back.

The MR functions that Aster provides, and the functions we can create on our own, make our analytics even faster. And having Scala programmers on our team, who can help write MRs so that we can do things Aster can’t do out of the box, is really powerful.

Which technology do you personally use the most?

Most of my work involves working with SQL, but also Cognos and Looker. We do a lot of data mining and looking at large sets of data to find patterns.

We’ve been a bit bullish about Looker since we started using it. What are its advantages?

Looker has empowered people across the business to do their own analysis. It’s generated a collective sense of ownership when it comes to our data and analytics. The more powerful self-service is, the more insightful our analytics can be, and the more the data team is freed up from mundane tasks involving pulling data for people. We’re free to discover new things and take on new projects, not just build reports for the rest of the company.

What are your favorite things about your job?

Interacting with all the different teams—it helps me to understand the business more. Designing new solutions and participating in the discussions on how to best approach an analysis, choose the data to work with. I also appreciate the diversity of projects. Usually the data team will focus on helping one department at a time for 2-3 months, but even then we’re still working with everyone.

What do you like most about the data team’s culture?

Everyone is pretty independent, yet we’re pretty close. We’re all there for each other, but we give each other a lot of leeway to build and develop the way we want to.  

And what about Gilt’s culture in general?

We’re quick in getting things done.  Everyone is really helpful.  We act on new ideas quickly and we’re not afraid to fail.

Earlier today the folks behind Gatling announced the release of Gatling 2.0.0-RC1—and also their new website, which mentions Gilt as a user! In case you’re not familiar, Gatling is an open source load-testing framework based on Scala, Akka and Netty. Congrats to the Gatling team for both achievements. 

image

August 8-9 marks the first-ever Scala By the Bay conference: two days of training, talks and reverie for the San Francisco-area Scala and functional programming community. Scala By the Bay is made possible by Alexy Khrabrov, conference co-organizer (along with Jason Swartz) and founder. Alexy’s also the founder and co-organizer (along with Swartz) of SF Scala, whose +2K members “use Scala to dominate the world.” Exciting! We chatted with Alexy to find out more about next week’s conference—try to go, if you can!

Compared to last year’s Silicon Valley Scala Symposium, which you also organized—and have rebranded as SbtB—what’s different and new about this year’s conference?

This year everything is different. We renamed the conference Scala By the Bay to attract folks from all over the world to our beautiful area. We’ve rented Fort Mason for the location and are holding the conference over two days instead of one. We ran a proper call for proposals, and selected a very competitive program. We decided to do just one track to make the conference more communal, and have a hackathon and an unconference scheduled for the evenings.  

You’re also offering training this year.

Yes, Scala and Spark training sessions; both are filling up rapidly.  My company, By the Bay LLC, is a Typesafe Training Partner, and we offer Spark training jointly with Databricks. After the conference I’ll do training regularly.

In terms of topics in Scala, what’s new or different about this year’s conference?

The Big Data/Scala theme is even more powerful this year, with a separate keynote on Spark in Scala given by Databricks CTO Matei Zaharia, who wrote the original Spark in Scala. The other keynote will be by Marius Eriksen, whose work at Twitter has demonstrated how Scala can power web-scale companies in real time.

How many people submitted talks?

We received nearly 50 proposals for about half that number of speaking slots. Some of the proposals we compressed to halftime. It’s still plenty of time and much more informative than lightning talks, and lets folks get a taste of the tools and approaches (which are hard to pick up on your own, or not as exciting).  We hope folks who didn’t get into the main program will propose their talks to the unconference.

How many proposals came from women?

We do have talks from women, but should work more on the outreach.  I’m personally following up on it, and this work started to pay off.

Did you notice any patterns, trends or similarity topics-wise?

Big Data is ascendant, with Spark a darling of the whole Big Data community at large.  We see people registering to learn Spark, which they then realize is in Scala, and so they join SF Scala next and ramp up on Scala itself. Akka is a perennial favorite, with not one but two proposals showing how to run farms of R servers with it—building a SparkR at home, so to speak. Various aspects of Play and web applications are dominating, showing how Scala-based web apps are taking hold.

Who is traveling the longest distance to speak?

Someone is coming from Hungary, and another from Argentina. 

What surprised you about this year’s proposals?

The sheer strength and depth.  Also the amount of type-related work, which got many votes. Scala folks are intellectually rigorous and efficient at the same time, and that’s why we’ll eventually take over the world.

How has the SF/Bay Area Scala community grown/changed/evolved over the past year? What would a visitor expect to find?

Play, Akka and Spark are growing exponentially and bringing more and more Scala beginners to the community.  The importance of professional training is obvious, hence our offerings.  More startups are relying on Scala, and whole categories of businesses are aligning around it, such as healthcare-related API startups. 

If I’m a Java programmer who’s new to Scala, will I be able to follow along with the talks?

You’ll have a much better time if you begin by taking Fast Track to Scala with Brendan McAdams on August 6-7. Brendan is a bona fide Unix/Scala/Akka longbeard, and he authored both the original and reactive Scala drivers for MongoDB. But our Scala speakers are excellent communicators, so every level of ability will be able to advance to the next one.  See you shortly By the Bay!

Here’s Typesafe Senior Software Engineer/Scala In Depth author Josh Suereth earlier this week at Gilt’s NYC office, teaching his sbt workshop. Josh spent more than two hours covering settings, configurations, dependency management, and other topics. More than 50 non-Gilt technologists from the metro NYC area joined members of our engineering team for dinner and the class.

If you’re curious about sbt but couldn’t attend the workshop, check out Josh’s presentation from the 2014 Northeast Scala Symposium: