Five Things They Don’t Tell You About StartupBus

One year ago, Jacek and I got on the NYC StartupBus and started Exversion. The ride didn’t stop once we got to Austin, but continued to an RV in SF, VIP parties in Washington, TechCrunch in Berlin, and keeps on rolling despite the blog silences as of late.

A month or two ago a VC told us to stop telling people we started this company on a bus. “It makes it sound easy,” he said. “If this is something that could be built on a bus then what’s to stop anyone from building one.”

What he didn’t understand, but you soon will 2014 Buspreneurs, is that StartupBus works hard to make sure the best possible candidates get on board and that alumni continue to support and help each other out. That’s what will make this trip one of the most insane, most amazing experiences you’ll ever have as a hacker.

But as you get ready to go, here are some things they probably haven’t told you:

1) Jailbreak your phone
Yes, there is Wifi on the bus. But it will be AWFUL. While your conductor and some of your busmates will probably bring Mifis with them, don’t leave things to chance: Unlimited data and jailbreak are your path to happiness. (Since Jailbreaking is um… illegal thanks to the Register of Copyrights at the Library of Congress, we don’t recommend you do it, but for purely educational reasons you should check out – http://evasi0n.com/)

2) Steal toilet paper everywhere you go
Getting stops out of your conductor will be a challenge, trust me. You’ll also eat at a lot of Walmarts. Subway is the StartupBus equivalent of the farmer’s market, you get it? And your conductor will try to make you believe that it is totally reasonable to restrict motion of the ocean to fifteen minute opportunities here and there, but there’s a bathroom on the bus for a reason.

3) Pack for three days
You’ll get so much swag from startups at SXSW, you’ll need the extra room in your luggage. Although, I don’t know many startups that give out branded underwear, so maybe stock up on that.

4) Half of the people on your bus will seem crazy, only a couple will be literally crazy
Not everyone who signed up will want to start a company. Some will be looking for adventure, some just want a whole bunch of people to hit SXSW with. Group think, sleep deprivation, and the extremeness of the road trip will enable even the most rational. It might all seem reckless and completely ridiculous, but you’ll bond in this bootcamp. When you get back to NYC (or where ever) you’ll have thirty new best friends.

5) Not everyone will stay committed, but it can work
A year later Jacek and I are still building. I’m still being flown around the world as the CTO of Exversion. StartupBus doesn’t have a shiny list of companies that actually became companies, but it happens more often than you might think. Your best chance at the post-Bus postscript is to find people you really enjoy working with. On the first day people will join groups based on what idea sounds the least likely to suck on day three and with people least likely to annoy the f*ck out of them along the way. Boot up your lotus position and do some koans because those two factors are not a recipe for a great team on their own. You will have to accept the reality that people will come and go. More to the point, you will have to let some people go. Who people are when they’re stuck on a bus together is not necessarily who they are when they get back into their routine at home.

Best of luck and safe journey to our 2014 Buspreneurs. Looking forward to seeing you guys in Austin!

The Ethical Hackathon or: How we learned to put on a good hackathon and make hackers happy

It used to be that hackathons were organized by hackers, for hackers. Prizes, if there were any at all, were small. People played more for the glory and the challenge. Plenty of teams formed adhoc, more experienced hackers taking less experienced ones under their wing. A hackathon was a place where you brought that crazy hack you always wanted to try but just never had the time to experiment with. Instead of sitting by yourself on a Saturday, you sat down and worked on it with like minded people … over a couple of beers perhaps, sometimes late into the night.

This wasn’t a utopia that exists only in retrospect, my second hackathon ever our team stopped what we were doing for the better part of an hour to give an aspiring programmer, a woman of color, tips on books to read and languages to learn. As a clueless newbie hackathons helped me build relationships with technical mentors in what is sometimes a closed off, less-than-inclusive community.

A hackathon used to be a nice way to spend a weekend. Networking, socializing, and building … I used to look forward to them.

Now, of course, hackathons have become big business. At first the corporate component was unobtrusive – free food in return for a product demo – but more and more hackathons are being organized by people who have never actually hacked anything themselves. They pull the basic format, pump up the prize money, have legal add a whole bunch of rules to suit their agenda and expect the best and brightest to give up their weekend to work for free.

And as a result not only are hackathons not fun anymore, they’re also becoming rather exploitative.

Reclaiming the Hackathon

Feasting on Data

A year ago I heard about a little hackathon being held in the middle of a high-powered social good conference called The Feast. Hackers who signed up got full access to the conference and its many perks. It sounded like a good deal to me, so I signed up.

The challenge of The Feast hackathon is building things with data. At the time I was just coming to terms with the gap between the possibilities of data and the reality. Participating helped me articulate everything that bothered me with the current data infrastructure. Instead of doing a traditional hackathon demo, I ended up using my time on the main conference stage giving a mini lesson in why hackathon ideas never become real life solutions (and also, a tutorial on brute force hacking for newbies)

(my part starts at 7:44)

A big target for me in that speech was The Khan Academy‘s API, which gives you full access to all the Khan Academy’s educational resources … with one catch.

There’s no good way to search for content. So in order to get what you need, you have to know exactly what you need and the exactly where it is. Oh sure, there’s a topic tree, but you can’t query it. So in order to get that information you have to download and parse the entire json file … all 30MBs of it.

This year Exversion was approached by The Feast to provide data for the hackathon, so I set about fixing the Khan Academy problem. It took me some time to figure out how best to untangle a tree when each branch was an infinite number of levels deep, but once I did I was able to produce a query-able directory of The Khan Academy’s content. Why should you have to sort through all the economics content when what you want is videos on basic addition? Now you can grab your Exversion API key and do this:

https://exversion.com/api/v1/dataset/3QC5N5HXIGSJK3Z?key=xxx&title=addition

You can even use complex queries to search the videos themselves, Check out our API documentation for more details.

Differential Diagnosis in Production : The Outcome

(At last the conclusion to Differential Diagnosis in Production. Could you figure out what was wrong with the server? Here’s the answer…)

As panic began to set in, I remembered something useful. Some commands on Symfony’s console are more picky than others. They will not execute if there’s an error, even if that error is something small in a file that has nothing to do with what the command should be doing. In the past I found this strictness super annoying, but now I realized it could be useful. If the server had stopped writing errors to the logs, than perhaps I could force it to write the errors to standard output instead. It was a long shot, but I didn’t seem to have any other options.

And it worked.

I chose php app/console doctrine:schema:update --force, which checks that the database matches the ORM schema outlined in the code. Since there were no updates, the script should run and report back that everything was up-to-date. But instead out came out a big red error block.

Son of a– We were out of disk space.

“That’s impossible,” Jacek said. “We have like 80GBs.”

And yet pulling up the disk usage confirmed it, the web root was using over 75GB of space. Suddenly everything made sense. The problem had nothing to do with Symfony’s cache, but clearing the cache opened up enough space to allow Exversion to operate. And when the server didn’t even have enough space to write to the logs, deleting the cache stopped fixing the problem.

So one question remained: Symfony doesn’t use that much space. Our entire development environment, all components of it, is only about 2GB. What was eating up all that space?

In a word: downloads.

How Do People Want Their Data?
Bulk data downloads were never part of our ultimate vision, but we had to provide them in order to give people the ability to branch and remap the data they wanted. It was an inefficient solution, but until we finished the features that allowed people to edit their data through Exversion directly (and we’re almost there) we had to offer it.

What we didn’t anticipate is how popular downloading data would be. And why would we? All of our competitors focus on serving files. They are older, more established, larger companies with greater visibility. Everyone who comes to Exversion has had experience with them.

So we assumed that what people wanted from us were features that our competition didn’t offer. After all, why download a file through us when you can go directly to the source and download it there?

Part of the answer is that people clean data found on those other sites and upload it to Exversion because they can’t upload them on our competitors. Another reason is that we index all those other sites for unified search. There are some who would say no one needs data search because Google does such a good job indexing everything … and those people have clearly never actually tried to find data through Google. It’s hard. It’s really freaking hard. We have three upcoming events where we’ve been given the rank and privileges of sponsors just for helping the organizers find data.

But back to our technical troubles, Exversion doesn’t host files. Downloads are generated dynamically based on the current state of the data. To keep the load on the database down, we don’t delete the file generated right away but clear them out on a regular basis. All that was working fine. We just didn’t anticipate how popular downloading would be. To put it simply: we weren’t clearing out downloads often enough.

Earthquakes, Drones and Gun Violence
Once we realized what the problem was, our first assumption was that this was all about PLUTO. PLUTO (for those that don’t know) is a massive collection of real estate data split into two parts: metadata and shape files. We released PLUTO, including a GeoJson version of those shape files, about a month ago. Obviously downloads were eating up more space faster because people were downloading these much larger files.

….Nope.

This is what people have been downloading

There were over 4,000 downloads between the regular clearing of files and the crash. And hundreds more have been downloaded since Labor Day. The most popular of the MapPluto datasets was Queens, followed by Staten Island of all things. Manhattan is the least popular.

Then there are the earthquake datasets, which are overwhelmingly popular both on Exversion and on Data.gov where they originate. I have no idea what people use them for.

Do We Change What We’re Doing?
Month ago I wrote a somewhat contraversial post about cognitive biases in tech. Some interpreted it as a criticism of Paul Gramham and YC, but if it was it was even more so a criticism of myself. The post spent much more time detailing false assumptions that I had made … and apparently continue to make.

Exversion had a system for reporting the number of downloads. There was a bug in it, which was why we didn’t notice the escalating level of activity until it took us down. I actually had noticed the bug months before all this, but I did not make fixing it a priority because it confirmed what I already wanted to believe about what we were doing. In other words, fixing the download counter seemed less important than handling other things because I assumed that users weren’t as interested in downloading data as they were in other features … but as long as the download counter was broken we couldn’t collect the data that proved me wrong. That obvious fact never occurred to me … which is ironic really.

So, once discovered, the problem was easy to fix: I cleared out the downloads, rewrote the process of generating downloads to be more resource efficient, fixed the download counter (adding some extra logging of that activity to boot) and tweaked the cron job that clears downloads to clean up that folder more often. For the time being we’ve disabled downloading of our much larger Pluto files, because we have to figure out how this changes things for us.

I built the download system based on the idea that it was a temporary solution that would ultimately be phased out. It was neither designed nor intended for the type of usage its seeing. I’m not sure what its performance is like on those large repositories, to be honest.

And since we are getting a lot of activity in this area the question remains, do we need to rethink our philosophy about the future of data? Should we make downloads more of a priority and redirect resources to building better infrastructure there? Plenty in Silicon Valley would say “Yes, absolutely”, but if the world was run by venture capitalists we would have more apps for Miley Cyrus to humiliate herself on and no roads. Building infrastructure is an unglamorous thing. It’s value can’t be gauged with public opinion, because it’s value doesn’t appear until someone builds something on top of it.

Lately we’ve been having more and more meetings with NGOs looking for a way to share and collaborate data. The feedback we’ve gotten from them suggests we’re on the right track. There’s a need that no one is serving that we are uniquely positioned to fill … and once implemented the solution could be truly game changing.

So we’ll see.

Differential Diagnosis in Production

This week we celebrated Labor Day by giving our web server some time off 🙂 Why should the relaxing qualities of holidays be reserved for conscious life forms only? One day the machines will rule us all, might as well start making nice now.

But seriously, the process of diagnosing and debugging a critical failure is not something often talked about, yet every technologist will have to deal with it eventually. Even if every piece of code you push is absolutely perfect, modern day web programming has too many moving parts … you can’t be an expert on all of them.

And when your service is down, the clock is ticking, and the problem appears to be buried deep in something beyond your expertise, everything depends on your resourcefulness. So, yes, this weekend Exversion suffered a critical failure that took the site offline for about twenty minutes on Sunday and about two hours on Monday, but the process of tracking and eliminating the problem turned out to be pretty interesting.

Early Symptoms
It started with an email from Lee Byron. Mixed in with some general feedback was an offhand comment about difficulty logging in. The login seemed to be “broken” but no other details were provided.

Neither Jacek or I were able to reproduce the issue, nor were any of our other users complaining. I emailed Lee back anyway, asking for more information about the error he got (if any). At this point there seemed like two likely explanations:

• Some strange cross browser issue.
• He hadn’t activated his account by verifying his email address

When I didn’t hear back I assumed the issue had been “user error” and took it off my agenda.

Locked Out
A few days later on Saturday morning, I was giving a demo to a perspective client on his computer. I logged in, only it didn’t take. No errors, everything appeared to work, but the “Login/Sign Up” buttons were still on the top nav and the members only features weren’t displaying.

“Do you have any kind of privacy, anonymous surfing software installed on this machine?” I asked.

“No, not that I know of.”

Exversion is built on Symfony2, using Friends of Symfony’s UserBundle (with some custom tweaks to the invitation recipe to suit our purposes) for User Management. By default this stores sessions in Symfony’s cache.

So the first thing I did when I encountered this problem myself was SSH into the server and clear the cache.

I could then login without any problems, and the rest of the meeting went really well. But obviously this wasn’t a real solution. Why wasn’t Symfony creating user sessions correctly? Was the problem with Symfony itself or FOSUserBundle? I started googling and digging through github issue reports, trying to find a clue … but there was nothing.

Original Diagnosis
There also wasn’t anything useful in either Symfony’s logs or apache’s so the best explanation I could come up with on virtually no information was that old sessions weren’t being cleared away and were filling up the cache. The solution seemed to be to move sessions out of the cache itself, something I had been meaning to do for a while anyway because I push code to Exversion every day and every time I clear the cache it logs everyone out.

But I wasn’t familiar with the procedure for moving the sessions folder and I wanted to read up on it and run some tests before going through with it. Like most, I prefer to do major changes to the infrastructure somewhere around 3am~4am so that if something goes wrong (God forbid) the impact to users is minimum.

And unfortunately I had to be up early on Sunday to head out to my parents. I hadn’t visited my family in about a good two months … pretty bad seeing as they live basically an hour away. So I figured I would keep an eye on the server and leave the permanent fix until I was back in the office on Tuesday.

Offline (part one)
I was in the middle of having a mojito with my mother and looking at pictures of their cruise to Alaska when I got the email from Jacek:

Not to ruin your weekend, but can’t login and can’t register on the site

login redirects to splash page, and register gives me a 503.

Effin’ hell … already? “Okay,” I texted back. “It’s a problem with the cache, I’ll fix it. Don’t worry.”

I SSHed into the server and cleared the cache. Then I reloaded Exversion.com expecting to be able to login and go back to my drink. Only what I got instead was a 500 error … and not one of our pretty custom ones either.

The site was dead.

HOLY CRAP, Fix It Now!
I spent the better part of the next twenty minutes texting back and forth with Jacek. He had limited wifi at his apartment and couldn’t get into the server but was able to confirm that the server itself was still running. The worst part of dealing with these sort of things is that Exversion is at present a two person team. I write the code and design most of the infrastructure, Jacek maintains the servers and does the UX. When something unexpected has gone wrong the best asset is another perspective or (barring that) the ability to step away from the problem and clear your mind. But we needed to get back online ASAP, and with only two of us fresh perspectives and breaks were in short supply.

I went through the usual rounds of magic tricks: I restarted apache, I tried clearing the cache again. Symfony’s logs now had some useful errors in them announcing that Twig could not write to the cache folder.

99.9999% of the time this means the permissions haven’t been set correctly. The cache folder is owned by the root user, but apache needs to be able to write to it.

And this made absolutely no sense because the permissions had not been changed and Exversion had been functioning correctly with those settings for months. How could a folder change its own permissions magically? It couldn’t.

But I didn’t have any better ideas, so I tried resetting the permissions. No luck. Examined the subfolders to make sure the command had executed recursively. No dice. Finally I just got so  frustrated I deleted the entire cache folder, recreated it, reset its permissions and reloaded the page.

Bingo. We were back up.

At this point let me stop and quickly explain something for people not familiar with Symfony. Symfony has a neat set of command line functions that help with a variety of administrative tasks. The cache is separated out into multiple folders for each environment (although most applications will only have two: development and production). The command to clear the cache requires you to specify which environment to clear, and it won’t touch the other folders in the cache. If you specify no environment, it will clear development by default.

In addition, Symfony rebuilds its cache when it detects something missing. So deleting the entire cache directory and recreating it wasn’t as extreme as it might sound. Symfony would rebuild the folder structure it needed automatically.

Offline (part two)
I ran a few tests, confirmed the site was back online and back to fully functioning, then I logged on to Exversion’s official Twitter account and announced that we were working through a problem with Symfony’s cache and to let us know about any errors.

I went back to google. It no longer seemed likely that this was a problem with user sessions. Could it be some bug in an update to Symfony core? Could it be an apache issue? I was having a hard time identifying anything useful and yet I knew that it was only a matter of time before the problem struck again.

It happened about 3:30 the next afternoon

@exversiondata Hi! I’m getting HTTP 500’s on everything but your landing page right now… Happy Labor Day!

— Aaron Schumacher (@planarrowspace) September 2, 2013

I confirmed the issue myself and tried deleting the cache again, hoping another rebuild would work the same magic.

No luck.

The whole site was down again. Now not even the front page would load. To make matters worse the logs were not updating either. The server log was cut off midline, which I had never seen before, while Symfony’s log wasn’t updating with any errors.

I was flying blind.

About a month ago, just before we came out of alpha, Swift Mailer stopped sending emails. Since we were in private alpha, it wasn’t a major tragedy, but– like now– there weren’t any errors being thrown. I spent the whole day working through every component of our email system, until I figured out it was a bad update. I tend not to expect large, widely used open source projects to fail … but as soon as I updated the package the problem was solved.

So now with no other options, I ran a composer update and prayed.

Nothing.

Exversion was broken and I had no idea what was wrong or how to fix it.

(To Be Continued. Part Two with the solution will be posted on Sunday. For now, take your best guess. What troubles this ailing server?)

Developers, developers, developers — and the first Exversion Ruby Library

While the video of an excited Steve Balmer shouting “Developers, Developers, Developers” has gone down in history of MSFT and software as a somewhat interesting moment, the man was right in that developers, simply put, rock.

As developers ourselves, we set out building Exversion on the Startup Bus without any plans for it outside of solving our own incessant pain points — those of easily managing open, remapped, and transformed data for use in our own applications — but the response from the developer community at SXSW and later on in NYC and SF made us acknowledge that we may in fact be onto something.

And since launching the closed alpha a bit over two weeks ago, a couple of really cool things happened. The first came in the form of our very own community-built ruby library for accessing the Exversion API. It was built by @bswinnerton and you can grab it here on GitHub. Thanks a million, Brooks!

The Sad State of Open Data

While the wealth of applications built on top of open data is ever increasing, a fundamental problem persists in  surfing open data websites. In short you could say that the entire process is nothing short of  a sad affair.

For example, take a complete listing of every farmer’s market in the United States, that includes the types of produce one can typically find there, or demographics on the Martial Status of Active Military Personnel. The variety is simply astounding, and while these are only two examples, there is just so much data out there. However getting to those bits of good data can be quite a pain.

People who love data see how gleaming insight from even static sets can help us hack everything from public policy to our daily commute. Unfortunately in order for developers to build these tools it’s not enough for data to be open and public, it also has to be accessible and frequently it isn’t.