- Adam Keys (twitter github therealadam.com)
- Charles Max Wood (twitter github Teach Me To Code)
- James Edward Gray (blog twitter github)
- Josh Susser (twitter github blog)
- Ruby on Rails
- Amazon EC2
- Deploy several times a day
- Anyone with commit rights can `cap deploy production`
- Feature toggles
- James Golick’s rollout gem
- Pride in your work
- Empathy for users
- Feature branches
- Adam’s blog posts
- Minimum Viable Product
- Streamlining your feature set
- Double writes and dark reads
- New Relic
- Eager Loading
- “Log profusely and get handy with grep, sed, and awk”
- Quicklook (Josh)
- qlmarkdown (Josh)
- Occupy Wall Street (Josh)
- Learn of the command line tools in linux (James)
- grep (James)
- bc (James)
- bc article (James)
- xxd (James)
- command line magic on Twitter (James)
- backbone.js (Chuck)
- Apache Software Foundation (Chuck)
- Classic Albums – Stevie Wonder: Songs in the Key of Life (Adam)
- The Promise (Adam)
- EconTalk (Adam)
- NPR: Planet Money (Adam)
- Evan Weaver’s Distributed Systems Primer (Adam)
- Java Concurrency in Practice (Adam)
- iTunes U: Econ 113 from UC Berkeley (James)
ADAM: So how does this… can you give me cliffnote’s version of the protocols, and how this works?
JAMES: Yeah. So far, you are doing it wrong.
CHUCK: Hey everybody, and welcome back to Episode 29 of the Ruby Rogues podcast this week on our panel, we have Adam Keys from Gowalla.
ADAM: Hello everybody.
CHUCK: Do you wanna introduce yourself real quick?
ADAM: Hey everyone, Adam Keys. I am an infrastructure expert typist dude at Gowalla. I’ve been doing Ruby for several years and coding for… I don’t know, ten or twelve years. I’m totally [unintelligible] or something. I don’t know how people say how many years they’ve been coding.
CHUCK: [Chuckles] I don’t know. We also have James Edward Gray.
JAMES: Hey everybody, good to see you. I’m not a dude, I don’t think. And I’ve also never impersonated the entire crew of Star Trek enterprise. Just FYI.
CHUCK: [Chuckles] Yeah, James’s pick this week is going to be eyes. All right, we also have Josh Susser.
JOSH: Hey, good morning everybody. That’s it for me. [Chuckles]
CHUCK: [Chuckles] Okay. I’m Charles Max Wood from teachmetocode.com. And yeah, let’s get this rolling. So what we are talking about today is keeping up with Gowalla. And we are talking about kind of their infrastructure and things like that. So, what do you do at Gowalla that’s interesting, Adam? I mean, other than tell people that they’ve been somewhere.
JOSH: Wait, wait, we are talking about Gowalla? I thought we were going to talk about rock music.
JOSH: Isn’t that the whole reason to have Adam on the show?
JAMES: It is a very good reason.
ADAM: We could talk about rock music. But I´ll probably take it in soul and jazz too.
JOSH: Well, that’s okay. I just wanna know what I should be listening to.
ADAM: You should be listening to Stevie Wonder. But I´ll talk about that during the picks.
JOSH: Oh great, I can wait.
ADAM: So the priors on Gowalla, it’s a social network for sharing stories about where you’ve been and sharing interesting things that you’ve found out in the world. So, I found a great burger joint, these are the places you need to go get tacos in Austin, etc. From a technical perspective, we’ve done a really shop from the start, the co-founders Josh Williams and Scott Raymond, did one of the first commercial Rails [unintelligible] sale way back in the day. I don’t know how many people remember that guy. And they did a Facebook game called [unintelligible] and then Gowalla; those, then Ruby apps. Gowalla started life as an app and when I came on, we converted it to Rails 3, right at the beginning of 2010. So, right before it started getting into betas. And we chose to do that because I chatted with Yehuda, and he said it would make him very sad if we didn’t go with Rails 3 instead of Rails 2 — so we did.
The interesting bits, the architectural bits, Ruby on Rails, we run on EC2, we use PostgreSQL as our main database, we use memcached, we use Redis and resque respectively. We use Solr a little bit for doing our geographic searches. And just lately, I’ve been working with Cassandra and working that into our infrastructure, and moving a data that grows quickly or has really low latency needs intro Cassandra. So that’s kind of the 20,000 ft. perspective.
We deploy several times a day. Everyone can deploy, so if you are familiar with design, you’ve probably heard some of our designers, they can all push to the production site, which usually doesn’t have comical results. But every once in a while, we use them as escape goat. We push several times a day. We do a little bit TDD, not as much as I would like. And the great thing about the scale we are at, we see less than 10,000 request a minute measured by new relic. The great thing about the traffic level we are at is that, if something does go wrong, I push some software that’s wrong, then someone tweets about it within 15 minutes, then it’s easy for us to either roll forward and fix it, or just take it out and figure out what we did wrong and then come back to it.
CHUCK: So I have a question about that. Is Twitter less or more obnoxious than [unintelligible]?
ADAM: That’s a good question, because [unintelligible] can give you like false positives, but Twitter is very emo. [unintelligible] is not nearly as judgmental.
CHUCK: That’s true. But is really obnoxious. But I know Twitter can be too. So I wanna riff of something that you talked about there for a minute. I think I read an article, or listen to somebody talk about etsy and I think they do the same thing with Ruby on Rails, where they deploy several times a day. And they give everybody commit and push and whatever access. And anybody can basically do anything they can be deploy. How exactly does that work? Do you have some kind of CI that you run it through first or I mean, do you just push and that means deploy, or how does that work?
ADAM: So everyone who has an access to the git repo, you get their secure shell key and they can just do cap deploy production. We don’t run it through any… you know that’s as sophisticated as it is for us. And at first, when I joined at Gowalla, there were no tests at all, and people are still deploying like this, even the designers. Our support people can push copy changes, whatever. And at first, this completely terrified me. These thoughts of terrible things going on in my head. But one of the really useful things I learned is that everyone has been burned in a different way, by not being rigorous and pushing software. And so everyone has different scar tissue, and so it’s easy to like build up this like combinatory fear of pushing software. But in reality, most of the things that you are afraid can happen, either don’t happen or just aren’t that bad when they do happen. And so that was learning that, and being able to push forward with that was really liberating. But also, Gowalla doesn’t operate pace makers or nuclear power plants; like the worst we can do if we push something bad, is people can’t sign up or check in. And that’s a nice safety net to work with too.
JOSH: So I have a question about how you get set up for that. I’ve seen people who do continuous deployment in a couple different ways. And I like it. I think that when I worked on projects where you can do that, it’s awesome. But it seems like there is a lot of homework that you have to do to get to the point of where you feel comfortable, just being able to do cap deploy at any given moment. And you’ve been talking about that, but there’s also sort of quantum leaps that you have to make in your application architecture every now and then, that seems like if you just do cap deploy would be a little bit crazy. So can you talk about how you adopt your deployment strategy to be able to do things like shifting from MySQL to Cassandra.
ADAM: Okay, so another thing that we’ve borrowed from the Flicker and Etsy culture to accomplish that is feature toggles. So we have conditionals in the code that say, “If this feature is turned on, then do this query to MySQL. And if it’s not, then do it in Cassandra.” If the feature is turned on, do it in Cassandra, otherwise fallback to MySQL. We use James Golick’s rollout gem for that. And so that gives us the ability to do it on a per user basis. We can create groups and do it that basis. All of our users, user tables and we have a few admin, and then we have a street team elite, which is kind of our crowd source army of people who crack our database, so we can deploy it for them. We can deploy it, we can use roll out to deploy to a percentage of users, basically just modular on their user id.
And so, given those tools, it becomes plausible to incrementally roll out that kind of change. Like when I’ve been doing stuff to move timelines, move all our activity feeds, social data into Cassandra. The very first thing I did after I merged that was, I mean, I had feature toggle around at the conditional, and I turned it on for just me. So I pushed it, turned it on for me, and look to see if it worked. That way, only I ever knew if I totally botched it. And that’s a totally awesome… like that just really changes your game as far as thinking about pushing software, and getting courageous about pushing
JOSH: That’s great.
JAMES: So you would just complain to yourself on Twitter if it didn’t work?
ADAM: No, I have a system of lashings that I do upon myself, whenever I… no, I just laugh at myself.
JOSH: Do you use the mortification gem for that?
JAMES: That’s awesome.
CHUCK: So one thing that I’m been getting from this, and it’s something that I really… I guess, it just kind of occurred to me, that this continuous deployment approach, is kind of a cure for cowboy coding, because you can’t just toss something in there and throw it out there. If you are deploying several times a day, then if you are putting something in there, then it’s production code. It has to be. And it’s something that really appeals to me. How do you determine whether somebody that you trust to be deploying to production. And the other question is, have you ever let somebody go because they continuously break the build.
ADAM: We just basically made it if you wanna make a change, you can. And if you wanna make a change, and you know how to operate a command line, then you can deploy. We’ve never had to let anyone go or even sometimes allowed to be like, “Hey, maybe try this a little more before you run it.” But, I would actually say it doesn’t discourage cowboy coding much, because I know that we do a lot of cowboy coding, and we did develop a terminology for cowboy pushing at one point into the repo, but that didn’t really stick.
In general, if there is an error, then it’s on you to fix it. So if you push a syntax error, then you better be the guy fixing it. No one else wants to fix your syntax error, although that does occur. So I mean, I forget who has this principle that we sort of borrow, that our ops guy encourages us to follow this, “Do not push to master, and do not deploy and then walk away.”
JOSH: [Chuckles] So this is the question that I want to ask for a few minutes is, we’ve been talking around the technology and process for this stuff, what’s the culture that needs to be there to be able to support this operation.
ADAM: Yeah, so the culture I guess just needs to be pride in your work, empathy for users. So if something is just sitting there broken for a long time, then that’s not cool. So you have to be of the mindset that that is not cool. Someone you can tolerate broken things all around, then this approach probably is not a good thing for you to do.
What else culturally do we do… so when I started, it was very much a cowboy, get things done, just keep pushing and iterating even if you are iterating live. And sometimes, you have to iterate live, because it’s tricky to get data locally, all though there are some gems and tools that are promising to make that easier to reproduce things in your local development environment.
Especially in our last release, we’ve added a lot more process. Like now, we do feature branches and pull requests, and reviews on those pull requests. That actually helps a whole, whole lot. It didn’t really slow us down too much. When we started implementing process, I wanted to make sure that it didn’t slow us down because that’s one of the things that I think is special about Gowalla, is that we do move so fast and we are design-driven, so or visual, or not code design; the visual-design driven. So the designer wants to change some widget or thing seven times in a day, then there’s nothing culturally a process preventing us for doing that. Yeah, it’s mostly, pride in your work, being responsive to your… people who are watching Twitter and saying, “Oh hey, these like check-ins are broken or trips are broken, or we can’t do spot changes or whatever.”
And you know, we quickly triangulate on that. We are all in campfire all day, all the developers are in the office now. There’s no one remote so, there’s like, “Yo, this is not working.” Someone says, “Oh, I pushed something around there.” Or if everyone’s boggled and we all gather around in a dig in to it. And basically being responsive to the ops guy. The ops guy or the support person. It’s almost always going to be the first one to know something is broken. That person has a lot of to be like, “Hey, you need to stop what you are doing, and look at this.”
JAMES: So I just wanted to say I like what you said there about feature branches. And you know, I see like lots of different git strategies and scenarios. And I think it was GitHub recently I saw a talk from one of their guys, that showed he was like, “You know, we just do really simple thing; we just branch when we are making a new feature, we work on it in a branch, and when we are done, we put that back and master them. Masters are deployable thing.” And that like that really is complicated as you need to get. Just putting those features in other branches, then you work on them, and then when they are ready to go, you can bring them back into the fold. And I think that makes a big difference.
CHUCK: Well, it’s interesting too because it’s not just a collaboration tool. I mean, I have one project that I’m pretty much the developer on, and when I’m working on a feature, if I put it into a feature branch then my client comes to me and says, “Hey, there’s this bug that’s urgent, that needs to be fixed in the beta.” I can switch back to the master branch, branch off to fix the bug, come back, put it back in, and then get back to work on what I am working on, without having to deploy half-baked code.
JAMES: Yeah. I think that’s a good point. So Adam has written several blog post, and given a couple of talks on some of the stuff he’s talking about with us today and there’s some really good stuff in these posts. Definitely, I think a couple of them are required reading. We’ll put them in the show notes. But one of them, kind of along the lines of what we have been talking about so far is, relentless shipping, which is a blog post Adam wrote in August, which I thought that was really cool. It’s not very long, and it’s very succinct, but I thought it was great. Can you tell us the idea for Relentless Shipping, Adam?
ADAM: So, Relentless Shipping is kind of in reaction to two things I’ve seen kind of floating around; one thing I’ve noticed is that there are… in working with a lot of different developers, and watching a lot of software being built from a far and as part of a team, I’ve noticed that there’s amongst all the other type of people, there is perfectionists, and then there’s people who are extremely cut throat. And the perfectionists worry about every aspect of it, and are kind of weary to release something until it’s just right, and they are kind of worried that these three specific people on Twitter might not like it, or this specific demographic of users will be confused by it.
And in the other complete opposite end for the spectrum, there’s people who are just cut throat like, they all push stuff that like barely works or it doesn’t work at all, but they’re always moving forward. And somewhere in the middle, you doing relentless shipping. So the title is playing off of something, one of the GitHub guys wrote about relentless quality, about always trying to sharpen things, make it better, etc. and I’m all about that, but I worry that these people who tend more towards perfectionism will see that, and use it as an excuse to not ship in a shipping… In my opinion, shipping trumps almost everything. You can push something that barely works, get it out there, see if people like it, and then just keep iterating on it, and that’s going to give you something much better, and much closer to what you really need, than if you are to keep it in a laboratory and finely tune it.
JOSH: [Chuckles] So I don’t think a lot of people remember this, but that’s pretty much how Microsoft came to dominate the world. They would just put something together and ship it, and then let their users figure out how to make it better.
CHUCK: They still do that, don’t they?
JOSH: Well yeah, [Chuckles] but that’s how they built their empire; they had an idea, they didn’t care if it was crappy or what – they just shipped it and worked out the bugs later.
CHUCK: Yeah, that’s one thing too. I mean, it’s not just about having your users point out the bugs for you, or having them find the flaws on what you did, but ultimately, and this kind of harkens to the entrepreneurial movement Eric Ries…
ADAM: Lean Startups.
CHUCK: Lean Startups, right. Yeah, they tell you to ship early. And you know, in some cases, it’s not about the technical flaws, but it’s about the flaws in your thinking, as far as your business strategy. And if you put it out there, people will tell you not only, “This doesn’t work.” But they’ll also tell you…
JOSH: Why it doesn’t work. [Chuckles]
CHUCK: Yeah, “I don’t like this. I don’t need this, I need this, or why aren’t you focused on this over here, because that’s really my pain.” And you can get the feedback that is actually going to make your business go as well as the feedback that’s going to make your app work.
JAMES: One of the other great assets of it, in my opinion is that it forces you to sit down, and say what minimum part of this could I ship? You know, it’s amazing, whenever I sit down and write down a list of, I’m going to have this feature, I’m going to have this feature, and they are always these huge feature of stuff like that. Like, I’m going to put in a comment system, oh, I’m going to need moderation for that and oh, I´ll need to be able to have it where people can get emails when new comments are added if they need to, and stuff like that.
But you know, is that really the minimum you need to ship? You can put in comment system as we all know from the famous Rails, and 15 minutes video, that’s not that much work. It’s very simple, it’s something you can do. You know, you are not probably going to have your first bad comment that needs moderating on day one or a period time. And your users may have ideas about how they wanna handle moderation of comments or something, which is a discussion you can’t have, when there’s no comment system. So it’s almost like shipping starts the conversation. I think that’s important.
CHUCK: Yeah, and it’s the conversation that you have pretty frequently as a consultant with your clients; what is the minimum thing that you need?
ADAM: Yeah, the great thing about that, whether you are talking to the clients or people on your team is that you can reduce the problem space do drastically, when we were adding photos to Gowalla, I was working on part of the backend upload process, and I was like, you know, we were talking about the data models, you know, “Oh, we’ll probably need some kind of moderation.” And so we are going to need, basically a whole other system for moderating things, so people don’t just upload wang pics and then… and so there’s moderation, and people ought to be able to flag things as offensive or whatever. But in reality, we ship without that, and we still just do that manually. People say there’s an offensive picture at this URL. We’ll go and say, “Oh yeah, that’s terrible.” And delete it.
CHUCK: So I wanna jump in here, what mechanisms have you guys seen work, as far as streamlining your feature set? Figuring out what you need, and what you don’t. How do you do that?
ADAM: So one thing we do is we measure a ton. So we have an internal dashboard, where we can see how many people are signing up, how many people are checking in, and how many people are doing what sort of thing, how that’s trending. So recently, we made a fairly controversial amongst our user community decision to entirely drop one of the features that made Gowalla distinct, when Gowalla first came out and very distinct against the competitor that we are always lumped with. But it turned out that that particular feature was requiring a lot of policing; anytime we wanted to do something new, we had to figure out, “Oh, how does this affect items?”
We decided to remove the digital goods part, basically. So, some people were very into collecting and trading these items. But in doing so, just a few of them were severely abusing the problem, and kind of junking up our system and making the application less fun for everyone else. So we put in some code to measure how many people actually use items on a day to day basis. Found that it was a dwindingly small the of our users and made the tough decision to cut it. Besides that, I guess that, like I said, we are a design-driven shop. This very much thing start with our designers who are quite excellent. So features and evolution of features will start with a comp or something and kind of worked out way up from there.
But we also talked about cool things we could do. At one point, we had a feature where you leave a note for someone at a place, and so it would say, “Hey James, the next time you go to this restaurant, or the next time you go to this Ruby Conf, you’ll get a cute note from Adam.” And again, we built that out, we tested it amongst ourselves, we liked it, we built it into the clients. And then we measured how much people using it. And it turns out, it wasn’t plenty.
CHUCK: So cutting out a feature maybe that already exists. I think another interesting conversation is the one where you decide not to put in the moderation feature, or not to put in this other feature, where you cut it out before you ever build it. Is there a good process to go through for that? I know that each problem is a little bit unique, but is there a good way of identifying those things that maybe you shouldn’t put in the first go?
ADAM: For us, it comes down to scope. We’ll have some rough timeframe that we wanna ship something on, and we imagine a wonderful, beautiful, unicorn-filled world, and then we sit down and think, “Okay, now we actually design this thing, make it coherent, make it easy to use, and then build the thing.” And do the actual design work, the front end work, the backend work, set up the databases, etcetera. So usually, it just comes from scoping. And every once in a while, we’ll decide, “Oh, this feature clashes with this feature,” or “They are not orthogonal enough,” or “This feature confuses that feature,” so we’ll drop it in that case.
JAMES: So I wanna steer back to architecture a little bit — if I can for a while. Adam mentioned that he’s been working Cassandra in, and then we were also talking about the Relentless Shipping thing, where you build the little bit in there, and you get it in the infrastructure, and you go ahead and get it deployed and get it working. In one of your talks, which was mixing a persistence cocktail, I think you may have given that Rails Conf, but you had a tip in there that I have not heard before, and I really liked and that was, doing “double write and dark reads.” Can you mention what that is?
ADAM: Yeah, sure. So that’s another thing I like to attribute my ideas. So pretty sure that came out of Flicker. I bet Google does it too — did it beforehand as well. So idea here is you are bringing up a new database, or a new… or bringing up some kind of new system. You need to… it’s something where you are like, “I think I coded this right, but there’s probably edge cases that I am forgetting about or don’t even know about.” So, you need some way to bring the new system online while you are still using the old system.
So the way we gone about that is we start off by… we have some feature branch, where we just have two lines of code, where if you are going to create a check in, it says, “Save this to PostgreSQL” and then the next line says, “Also, save this to Cassandra if Cassandra is turned on”. So what that gets you is you can do real-life testing, with all the weird kinds of data people enter on your production site, but without having to worry about doing a big, giant leap cut over. You don’t have to have some boil of the ocean where you are like, “Okay, everything goes into the new database now.”
So once you start doing those double writes, and you’ve debugged all of them, then you start doing… once you finished doing the dark writes and you think they are all that code works, then you can start doing dark reads. So a dark read is this pulling from some database the size of your canonical one, maybe checking the result, and then throwing it away, returning the data from your primary database. So again, this is a way to make sure that your code interacts well with all the varieties of crazy data that you have in production. And both of these also lets you load test your new system; if you turn on double writes, and the new database falls over, then you just turn off the double write, and go back and fix it.
Same thing with the dark reads; you start turning on the dark reads, and you can’t sustain that throughput, then you go back, replan, push some new code and try again. And you know, you wrap this, it feature toggles, you continuously deploy it, all these things kind of play off each other. And one of the big things that struck me about how Kent beck explains extreme programming is that a lot of it is about giving the programmer courage to do really great things. And all of these things are kind of like the courage enablers – for me at least – in deploying software to production, and getting it out in front of everyone.
JOSH: That’s nice. I liked the way you said that. So I think part of what you are talking about is being able to tell the effect of your actions, being able to monitor how good is this software doing, or this change that I made doing. Can you talk a little bit about how you detect what’s going on in your software, in your systems?
ADAM: Oh yeah, I’m glad you brought that up because that’s definitely one of the pillars of making all these work. You have to be in your app, all the time. You have to be watching your performance monitor, so I stand several minutes a day at least, just looking at what our performance numbers are in new relic, where are we slow, what’s our slow queries, what can we improve. After I deploy software, especially if it’s a big thing, I´ll watch our performance numbers in New Relic like a hawk. So I watch that deploy marker, and if the graph goes up in to the right, then I start to get extremely nervous, pinging our ops guys saying, “Hey, does this look normal to you? Or did I totally botch something?” and he and I will go in and figure it out.
JOSH: Do you have any company-wide information radiators that keep this stuff in front of everybody’s face all the time?
ADAM: At various times, we have like a dashboard that’s more business metrics. And at one point, I used the New Relic API to integrate health checks in. So anyone in the company could go click this tab in our dash board, and see basically doomed faces. See you can a 100% health…
CHUCK: [Laughs] That is so awesome.
ADAM: Actually, I don’t think it was faces, it was our various avatars. At one point, our artists will do everyone on the company a sort semi cartoonish avatar, and they have different color background. So blue was fine, yellow was bad… and yellow was “look in to this”, and red was “this is on fire.” Oh, and I also put something at the bottom of the admin page, that would say, “Gowalla is feeling great.” “Gowalla is feeling ill,” or “Gowalla is on fire.” Because a lot of times, people will be like oh hey, are we down, “Oh hey, the Wi-Fi dropped out, and their phone is being stupid or something. So this way, people can and say, “Oh hey, Gowalla is on fire. I should type that into campfire and let everyone know.”
JOSH: That is awesome. I have not seen that before in an application. So I asked about the information radiator stuff, I don’t know if you’ve been in the Square office. Their office in San Francisco is pretty impressive and they had a couple of meet-ups there. And if you walk in, they have giant television monitors all over the place, that you just like look everywhere, and you can see status information about all of their systems, and how many transactions they are processing and how their applications are running, and loads. It’s really hard to be in that office, and not be aware of how everything is functioning.
ADAM: Yeah, one of the teams in our walls, we had a Graphite up, we were using a StatsD that Eric wrote, and so we were pushing trend data in there, so you could look at the wall, and see if some metric was down. And that’s a good canary in the coal mine. And we have that right by the basketball court that we constructed in our office, which is a whole other thing. So that had pretty high visibility. Unfortunately, that got claimed by the rules of entropy, but it’s a very useful thing. At some point, we should definitely bring that back. Having some kind of just trending data on what’s going on is super useful to figure out if you… if a product change you made is working or not, or if you botched something or do something well technically.
If people are checking in more, then it probably means… if people are checking in more and that thing in the product changes, then that means that we probably succeeded in making that code path faster, if more people are signing up, then we’ve made that sign up flow better. We also use a little bit of exception tracking. The problem with ours is that we got to the point where it becomes really noisy, really quickly, and we’re something like airbreak wasn’t keeping up with our error throughput. And at some point, we ended up just going with New Relic for exception tracking, and performance monitoring. And that’s actually worked for us pretty well. So like I said, after I deploy, I watch the request rate, and I also watch the exception rates.
CHUCK: So that’s interesting. I have to say first that it’s not fair to put that by the basketball court, because I would be walking over to the basketball court, see it and be like, “Oh crap.” [Chuckles] Instead of going and playing. Anyway, I’m a little bit curious about scaling, and about some of the challenges of scaling things out as big as you have. I did noticed that you are using things like Memcached, Cassandra and maybe Redis, they can all be used as kind of scaling things. And rest you obviously have to offload things that can be handled on the backend that the user doesn’t have to worry about being synchronous with their request. I’m a little curious though, how do these all fit in to the picture, and at what point do you realize that you need to add something like Cassandra or Memcached?
ADAM: So the first step is that our ops guy is watching our PostgreSQL database, and when someone pushes code, and all for a sudden the PostgreSQL database is not happy, he is pretty quick to go back and look at the commits and be like, “Hey, you did this thing in active record, and it’s probably generating a bad query. You should go look into this.” He’s also very good about keeping us honest about when to use either joins and when to just let that stay lazy. Reducing the number of queries is good, unless you pull back a crap ton of data that you only barely use. So definitely, our first step is only just optimize our queries… [cat noise] and keep our cats happy.
The next step is something that we just can’t get it to where we need it in PostgreSQL — which is our primary database — where you are going to cache it, … or some secondary representation of that, in something like Redis or Cassandra. And also, just using the right tool for the job; Solr is very good for doing faceted, flexible queries — far better than PostgreSQL — so use Solr for that. But it actually ends up that PostgreSQL to do just, “give me all the spots within 3 mile radius of this.” We have a crazy ‘order by’ in one of our queries that no one every touches, because it has a lot of crazy trigonometry in it. And that’s pretty quick. So yeah, you’d think that we’d be using like the PostGIS extensions or the geo stuff in Solr, which we do a lil bit more now, but we actually got a long way with PostgreSQL.
As far as moving to Cassandra, Scott had built a prototype of an activity feed service, and I got interested in Cassandra through that. And I built a training wheels project, something for a part of our data, we audit changes to our database. And that table in PostgreSQL is getting really big. So I wrote something to store all that in Cassandra, and that’s really a low-risk data. If we accidentally blow that away, then that’s not really big of a deal. If it’s unavailable for a minute or an hour, life goes on.
So that was a really good way for us to get familiar with Cassandra; decide if it was something we really wanted to use, or if it was more hyped than its actual quality bore out. And it turned out it was really cool, it has a very nice operational features, in that you can throw more machines at it, machines can fail overnight, and ops guy doesn’t have to get up — which I like a lot. And so, from there, we decided we start moving high velocity data into Cassandra. So we moved our activity feed stuff, because that data grows very quickly.
And then just recently, we had some data… we cached other social network, the social graphs that we fetched from them, so say Twitter or Facebook, we cached them within the appropriate time restrictions, so that we can say, “Oh hey, your friends on Gowalla from Facebook,” etcetera. And that data was growing quickly. We have that on the 16GB EC2 instance. It was getting to the point where it was growing very rapidly. So we spent a couple of days, we wrote that stuff in Cassandra, roll it out to ourselves first, that looks good, then we try to start to turn it on for 10%, 20%, 50%, 75% 100% of our users, and then we got Redis completely out of that equation.
So that’s more or less how we go about growing things. But we don’t start with something complicated; we start with Redis or PostgreSQL. We don’t go straight to Cassandra because that’s a little more involved and tricky for people to understand.
CHUCK: Right. I’ve been playing with Cassandra. I’m a little curious as to what you are using to connect with it. Which gems?
ADAM: I just use the Fauna Cassandra gem that Twitter started, and is kind of a stone suit kind of community effort at this point.
CHUCK: Okay, cool.
JAMES: All right, so I’m going to try to high jack again. In one of your articles, The Current and Future of Ruby, you kind of talked about the recent trend to kind of back to Unix, and object-oriented design in our community, how we seem to be, “Oh yeah, there were all these great ideas from the past. Maybe we should go learn from those.” And I too, noticed this trend and I’m really glad. But then, you go kind of deep in to one, the object-oriented design side in your article, “Your FriEnemy, the ORM,” which is absolutely one of my favorite articles that you wrote. So can you talk about why you said that the ORM was a frienemy?
JOSH: Before you get in to that, I’m going to have to interject here and do my typical Josh thing, and define frienemy.
ADAM: Yes! I was hoping you would say that.
So a frienemy, you’ve never been in high school with girls… guys can do this too, if you’ve never seen like mean girls or something, so a frienemy is someone who you either treat as a friend, but you kind of keep at arms lengths, it’s kind of like keeping… no, it’s not like that. So someone, or in this case, something, a tool, that is really useful up to some point, but if you take it too far or use it in weird ways, it will totally slap you in the face.
So in this article, I’m saying active record, — or any ORM really — is really great for getting your application off the ground. If I’m prototyping something, I don’t wanna think in terms of queries and updates and inserts and data structures; I wanna think in terms of the domain. And something like active record is really great for getting an application up off the ground, rapidly iterating it, and sort of building up a domain model, that everyone can talk about and share.
But at some point, it gets to where you are like, “Oh my god, I really hate these models that we had.” We have a junk drawer, user models that’s like 2,000 or 20,000 lines long, that has like every method in the system, or it becomes difficult to generate good queries, or things become too coupled or the objects aren’t cohesive enough. Basically, it’s either not object-oriented enough, or not scalable and operationable enough.
And so in this article, in your Frienemy, the ORM, I shared something that I’ve tinkered with, that I think I really borrowed from… the notion was, you have some object with an API, and then inside that object, you stash an inner class that’s the actual active record model. And the outer class exposes an API into that model. So, you can… an API, use active record where it’s strong, but then if you get to a point where active record is punching you in the face all the time, or just not serving your needs, you can rip that active record model out, replace it, and just change the coupling between your API class, and the underlying model.
JAMES: So this article — I got to stress — is amazing. And I’m going to required reading for everybody listening to this. He talked about over and over again, it’s really popular with like fast testing Rails cloud right now. I was at Ruby Midwest last week, and Uncle Bob’s keynote was basically on this topic about how Rails applications, the problem with them is they force you to think about them in terms of Rails applications. Like if you look at the directory structure of a Rails application, you see things like apps and controllers and stuff like that, and he showed us an example of one, and he’s like, so what is this app? And we had no idea. All we knew was that it was a Rails app, because you don’t know what the concern is.
And Adam talks about that in this article, about how if you focus on the ORM and those as the models, then it kind of forces it to being a top level concern — which gets you thinking about the wrong things. and what I love is the code in these articles is like super simple, in that, it’s really digestible, it’s not a huge blog post, it’s easy to go through, and he does things like swap out the testing structure to just use a trivial in-memory hash, instead of active record, so then he gets lightning fast test super easy. Or then he goes on to actually completely replace the database structure with minimal pain. Really awesome article. I have to stress – everybody needs to go read this.
ADAM: So I got a lot of the ideas by listening to this podcast — everything you guys have been saying. So if someone is just now coming on, I encourage to go listen to everything. Even if you don’t have a good time to do that, like Kent Beck does, it’s worth your time to go back and listen to all those stuff, because it’s definitely inspired by what all of you guys have been saying and writing about.
CHUCK: All right, then let’s go ahead and do the picks. Let’s start with Josh this time.
JOSH: Okay, thank you. So I have a simple developer tool pick, and that is for quick look. So if you are using Macintosh or Mac OS X, this is a great little feature where you can just select the file on the finder and hit the space bar, and it will zoom open a quick look view of the file. So we all do that with… it’s great to just like scan through a folder full of pictures or what have you. And if you have files that are not known to the system, they just come up as this black box. “Oh, here’s the file, great. Nothing you can do with that.” But Quicklook is actually an extensible system.
And I recently got tired of writing my blog post on a lot of documentation and mark down format, but Quicklook didn’t know about mark down. So, I went out and I found somebody had written a plugin for Quicklook to be able to parse mark down files, and suddenly now, I can just select a file, touch the spacebar and get to see what’s in the markdown file rendered using the markdown format. And not only that, but that allows the spotlight indexer to index the context of my markdown files now, so I can search for them using spotlight. So I´ll put a link to that plugin in the show notes. It’s up on GitHub. It’s called QL Markdown, if you wanna just look it up on GitHub.
If you’d start looking around, you can find that there’s this whole ecosystem of people who have written quick look plugins. So it’s worth checking out because as soon as you start down that road, you realize, “Oh hey, I can actually have a lot more of the data on my system accessible to me in an easy way.” So that’s my geeky pick.
And my other pick this week is to get sort of hyper political, is my pick is Occupy Wall Street. I live in San Francisco, there’s a lot of Occupy Wall Street going on here in San Francisco, across the Bay in Oakland, and some of the universities around town. I don’t live in New York city anymore, so I’m a little removed from the main action there, but we got plenty here. And I got to say, if you are living in a town where there is not an occupation action going, and you don’t know people who are participating in it directly, you probably have no idea of what is actually going on, because by enlarge, everything that’s reported in the media about Occupy Wall Street is completely off base. However, it’s not too hard to do some searching and googling online, and reading some blogs to see what’s going on. And it’s worth it to educate yourself about it. So I don’t wanna get too much into putting words in other people’s mouths, so I´ll just leave it at that.
CHUCK: All right, thanks Josh. James, what are your picks?
JAMES: So, when I was reading Adam’s talks and blog posts, he had this one line in his mixing a persistence cocktail that I loved, and it said, “Log profusely and get handy with grep, sed, and awk.” And I thought that was a great tip. I was late into my developing cycle, before I really learned how handy all of the Unix tools really are. Adam talks about that; moved back to Unix and how useful those tools are. And so that’s going to be my recommendation this time is to finally sit down, do that thing you’ve always promised yourself to do, and learn some of the command line tools that you have available to you. I know that we tend to put it off and just not do it — at least I did. But when you do sit down and learn them, it’s amazing how much more useful things come to you.
Like grep, that’s one of the things you should learn. I actually learned about this recently, but grep is a very powerful tool. You got to learn some of the options of it, like the ability to invert matches, or the ability to get context, and there’s before context and after context. And those are all extremely helpful when parsing log files. Like Rails log, you can once you find that person’s IP address, then you can grep for their IP address. And if you put some after context on it, then you are basically looking for their request. And that’s it. it’s just one command line, and you’re finding their request. So, grep is super powerful. And you got to figure that one out.
But there’s tons of other command line tools, that are awesome. Obviously, I can’t list them all, but just to give a couple other examples, bc is an awesome command line calculator that I use quite often, just pipe some Mac into it and it spits out an answer. It’s extremely handy. And the reason I bring that one up is there was an awesome article about it recently, kind of the gotchas on it. So if you wanna see why always give it the –l option, then this article will tell you. It’s really great. So that’s another one.
Another tool I’ve used that I don t think people appreciate enough is xxd. It has two absolutely awesome uses. When I was young, I used it to cheat in video games, because you can do a hex dump of the game file, and fiddle some bits and put it back using xxd, which is awesome. Nowadays, I use it to find encoding problems. So, if you are messing with some file that has a bad encoding, and something will open it, then I open it in xxd, so I can see exactly how it’s encoded and basically troubleshoot it. So that’s very helpful.
So those are just some tools obviously, your mileage will vary, and different tools will be useful to you. But seriously, sit down and learn some of them, because they are super valuable. And if you want to learn some of them, there is this awesome new Twitter account called command line magic. And the guy just tweets all these command line tips; how to put your SSH key up on a server, with just a single command line, or how to get an entire listing of the ASCII table – just these awesome command line tips. So everybody needs to go follow command line magic, because it’s just super awesome account. So those are my picks this for this time.
And it works pretty seamlessly with Ruby on Rails’ REST implementation. And at least in Rails 3, you by default, have the JSON API pretty much built in. So, it’s been really kind of cool to have it all work out. In Rails 3.0.x, you have to add a line to your configuration. I don’t have it in front of me at the moment, but basically, you have to tell it to not put it in its own little… nest it within its own object, and then and so if you wanna get the attributes, you have to reference in other key in order to get those values and it’s kind of a pain in the neck, but you can turn that off. I had to Google it. I found it on Stack Overflow, and I´ll put that on the show notes as well.
The other thing that I wanted to pick this week was the Apache software foundation. We talked a lot about the software that’s in there like Cassandra, and Solr, which is kind of front end API manager for Lucene. And this last week I know some of you are aware I went to Apache Con, which is their big conference, which kind of precluded me from going to Ruby Midwest. And the cool thing is they do a lot of things over there that is interesting. And I actually interviewed quite a few of their developers, and I´ll be putting that up on The Teach Me To Code podcast, including the software foundation president this year, and you know, kind of explain the apache way.
And I think there are some things that we can learn from them, that I think are important ways of building the community and managing projects. I don’t think we have to do everything their way, but I think there are a few things that I think we could learn from, and make sense for certain projects. But the other thing is they just put out incredible software, and I just wanted to give them a shout out, because they were extremely hospitable, and at the same time, they just do some awesome stuff in keeping this projects of like Cassandra, and Lucene and Solr and Hadoop, and some of these other things that are out there.
So anyway, those are my picks. And I also wanna just say that if you are looking at going to a conference that you travel to next year, I think that it’s important to get to Ruby one or two Ruby conferences, but at the same time, you should branch out and see if you can get to one of the other conferences out there, because there are a lot of lessons to learned from kind of getting that other point of View, the way that they deal with things in Java for example, was something that I got exposed a lot to at Apache con. And I think there are a lot of things that we can pick up from these other languages, and pick up from these other programmers who have all of these experience, that then we can bring back to Ruby, and just make our community that much better. So anyway, that’s what I’ve got. And we’ll go ahead and let Adam tell us what his picks are.
ADAM: Okay, so my first pick is a series of documentaries on pop albums, it’s called Classic Albums. The two that I really like are Steely Dan’s Aja, and Stevie Wonder’s songs in The Key of Life. And these are basically just, they break the album down song by song, talk the musicians or the producers they both are critics and talk about how the album was made, and what was each song about, where were these people in their life, what lead to them creating this particular album. And the really, really cool thing is sometimes, they will break it down, they’ll be the producer or the musicians and they’ll be sitting the mixing board, and they have the masters loaded on to the mixing board, and they’ll start some song, and they’ll isolate a track or bump up something that you would never have noticed about a song. And that’s really an interesting way to see how other created endeavors are making things, and see what they struggle with.
So what I love is to see how musicians struggle, and how that is similar to how we struggle with creating software; what are the similarities and differences, and what can I steal from how they’re problem solving in their domain. There’s also a really great one on the making of the darkness on the edge of town by Bruce Springsteen, it’s called the promise. It is this whole documentary about him, after like he made his first wildly successful album. Like it’s the pressure that he put on himself to make another album, and prove that he wasn’t just a flash in the pan. And he like he wrote like 70 or 80 something songs, and he only left 12 on the album.
And it took like two years, and there was music business politics going on, he was trying to get stuff done. And there’s one point that’s really great, where at some point, he decided he hated the way that the drum sounded, and so, he basically locked Max Weinberg, who was the drummer for Conan O’Brien but also Bruce Springsteen’s drummer, he basically locks him in a room, and made him hit his snare, moving his snare around in a the room, moving microphones around, until it didn’t sound like hitting a snare. So if you ever worked with someone who is a little too intense, then you’ll sympathize with that.
Another thing is a podcast on economics and finance. This is like something that I like to pay attention to because it kind of explains to me, one for the mechanisms of the world, like how the world works, but there’s a lot of good stuff here, especially how in the pure economics about the design systems, how system evolve, and how to observe systems. Basically economics is this topic where people… men and women get in a room and say, “We need to make the economy bigger. How do we do this?” And they usually fail spectacularly, because something happen… they make some rule, and then it has some unintended consequences. And that’s totally like working with software. You say, “Hey, you should never commit after 5PM.” And then you can’t fix your software after 5pm.
So there’s one podcast called Econ Talk, it’s quite in-depth, but very easy to… it’s not too jargony, there’s no deep math in it, and he covers a lot of topics. It goes a little astray from economics, but it’s very good as a deep dive, and the NPR Planet money podcast is usually a little shorter and more approachable. It was actually forked off of this American Life after they did on the financial crisis, so it’s very much in the this American Life style, but it’s all about explaining the world of money, and finance and economics and governance, and all these stuff. And so those are really great just for expanding your world view of designing systems and doing creative things.
Technically, the things that I’ve been to a lot lately is distributed systems and concurrent systems, so Evan Weaver has a distributed systems primer blog post, where he’s basically put together I think around a dozen academic papers on distributed databases, storage techniques communication techniques, design techniques how to deal with state, how to think about distributed systems. And reading several of these, I haven’t read them all, but they were all pretty easy to read and they all like just totally… you know open the whole new world of how to make systems. And it’s all very important for us as web developers, and Ruby and Rails developers because these days, almost every system we build is distributed in some manner. And I mean, if you have a web application in a database and they sit on… and they are not running in the same process, then it’s a distributed system. So, it’s useful to be aware of these issues, so that if you run in to them, you can recognize them, and not reinvent things poorly.
And then java concurrency and practice is something I’ve been reading. I think it’s one of the standard books on doing multi programming in java, and this I think is going to be very well to Ruby developers, as Rubinius and RVM… as Rubinius, Ruby 1.9 and JRuby become more prominent, and those get better concurrency stories, not so much knowing the java APIs, but knowing the concerns. Like one of the things I got from reading this book was that actually doing threads, and assigning work to different threads is actually pretty straightforward, but all the error cases are extremely gnarly, and make my head hurt.
And then you overlay locking with that, and waiting for locks and how to deal with timing out locks and how to structure the locks in your program if you have to do locks. But that’s just all new stuff to me; stuff that I wouldn’t normally call upon doing mostly Ruby and Rails web stuff that I do, so if you are like me and James, and you are really into this Unix and object-oriented renaissance, then learning the java concurrency stuff is a good primer, until someone writes an equivalent book for Ruby. And that’s it.
JOSH: [Chuckles] So Chuck said “picks” and Adam heard “epics”.
JAMES: [Chuckles] Hey, I got to add one to what Adam said. You guys know I love economic podcasts too, I’ve mentioned Freakonomics in the past, and Adam’s picks are excellent. One more amazing pick in that area is there’s iTunes U class economics 113 from UC Berkley, by Bradford DeLong and it was in the fall of 2008 is basically the economic history of the United States. So if you wanna learn history like you’ve never learned t before, you should go listen to that.
CHUCK: All right, well, thanks for your picks. Every time that Adam said “do locks,” I kept thinking of Shrek. Anyway, let’s go ahead and wrap this up. We are in iTunes if you wanna find us, you can find us there. You can find the show notes at rubyrogues.com. And we really appreciate the reviews that people have left. If you haven’t left us a review, by all means, go ahead and do what we… we had 50 odd reviews last time we looked, and we really, really appreciate the feedback.
I’ve actually been meeting people who are listeners of the podcast at different conferences and users groups, and it’s really, really nice to get that feedback that the people are listening to and liking it. So if you are enjoying it, then by all means, if you see one of us, let us know. And if there’s something else that you would like us to do, or some idea that you have for us, then by all means, let us know about that too.
Next week, we are going to be talking to Noel Rappin, we are still fine tuning on the topic, but he is the author Rails Test Prescriptions, and has lot of expertise there and in other areas. So, we are looking forward to that, and hope you all have a great Thanksgiving, and we’ll catch you next week.
JAMES: Thanks, Adam!
ADAM: Thanks, guys!