happy [data driven] holidays from EXVERSION

This holiday season we wish you all the best. Here’s a few fun facts about all things holiday.

1. The five most popular xmas carol words in order are christ, king, born, night, joy.

2. Rudolph is the most influential reindeer, having 8x the influence of Dasher, Dancer, Prancer, Vixen, Comet, Cupid, Donner and Blitzen.

3. Your best chance of a white holiday is in Northern Minnesota with a >90% chance of there being snow.

4. In 2013 Thanksgiving and Chanukah overlapped, the last time this happened was in 1918, except for Texas where it happened in 1945, and 1956. It will theoretically happen again in 2070.

5. The most expensive hotel to spend New Year’s Eve in is the Royal Suite at the Burj al Arab in the United Emirates at $72,000 including taxes. Ouch!

All the best, and again, happy holidays,

– EXVERSION

Going geospatial with Exversion

Image by Stamen Maps

Earlier this week we gave an impromptu and quick overview of Exversion at #NYC Beta‘s meetup. The majority of the talk revolved around some of the idiosyncrasies of PLUTO and MapPLUTO, and the audience, a largely geospatial crowd, wanted to know what GIS functionality if any we support.

While all we can say is that geospatial is dear to our hearts, at present all API output is for the time being in JSON. However, if the dataset contains latitude/longitude or x/y coordinates you should be able to use it with popular mapping libraries such as leaflet, and D3.js, as well as Google Maps, Bing Maps, et al., allowing you to map those JSON objects though our API.

An sample dataset that this would work with is one we featured during this years Publishing Hackathon, held during Book Expo America, Banned and Challenged Books.

When we run a simple search query on it, or look at the data preview on the dataset’s page, we see that it contains both latitude and longitude columns, along with other information about the challenged title, city, state, challenger, and other details.

The coordinates in the dataset, simply allow us to load a generic JSON layer, and display points on a map, such as in this Publishing Hackathon example by Jackon Lin who used the Banned and Challenged Books dataset in his visualization. *Displayed at the bottom of the page.

While this for the time being isn’t a complete answer to a GIS Data API, it’s a step in the right direction, and as we develop Exversion further, we hope to build in geospatial functionality that will make is easy, simple, and intuitive to import data hosted on the platform to a wide suite of geospatial data visualization tools.

And for the time being, if you build any apps, geo or other on the platform, we would love to see them. So please send your work to info @ exversion.com and we’ll try to feature as many of these as we can.

Now go click on that map and see what books people have tried to ban in the United States.

And we’re off, Exversion is now available to everyone.

original photo by NASA Goddard Photo and Video

We’re absolutely ecstatic to announce that today, August 7th, we’ve moved from alpha to beta, and as such, have opened the platform up to everyone.

Until today, data was stored in independent silos across the Internet and was often inaccessible. With this launch we’ve made over 40,000 datasets easily searchable from a number of sources, and will be adding additional data in the coming progressively moving forward.

While this data is now searchable, much of it remains unusable and we ask that the community help us in cleaning up the worlds data. With the platform you are now able to upload file of up to 10MB in your browser, but more importantly now also have access to upload much larger datasets programatically.

Every piece of NYC’s real estate data is now accessible through our API

This week we announced that The City of New York Primary Land Use Tax Lot Output (PLUTO) database is now machine readable. Less than a week after the City made the database publicly available, we’ve made all PLUTO data readily queryable and freely available via the Exversion API.

This means that city planners, community boards, researchers and other people seeking commercial and residential real estate data can quickly and easily search hundreds of thousands records.

Normally you’d have to pay the city for this data, clean it, upload it to your own server, now that’s it’s machine readable, anyone with an internet connection can instantaneously start deriving insight from this data, and we’re very excited to see what people do with this data, the types of application they build with it and what they’ll be able to uncover.

API Wrapper for Machine-Readable Data

As we crawl and index the world’s data silos, we’ve noticed that some have APIs, to this extent we’ve gone ahead and created an API wrapper for those data sets that are already accessible in a machine readable format.

The benefit of this is that you’re now able to build on top of both NYC data and Chicago data through the Exversion API without having to go to either city’s open data site.

While the functionality is there, some problems still persist, primarily that we’re limited to the API functionality of the host website, and henceforth some API functions may not be available.

For example, while you can query a data set housed on Exversion to give you all results that have a single word in a column, the same cannot be said for external data we’ve wrapped from places such as NYC, San Francisco, or Chicago, where you’ll need to query a full text.

Which is why we urge you to download the data set and upload it to the platform to take full advantage of our API, and you can do this easily, by clicking the Menu box and selecting download and then CSV, and then clicking the Green Upload button. When you do this, the meta, description, and data attribution fields will be automatically populated. Just add a blurb describing the changes (if any), and upload the file.

For the above data set, City of Chicago / Christmas Tree Drop of Locations, the entire process took approximately a minute, making the data readily consumable by anyone on the site, and when you or anyone else searches for the data on Exversion, because it’s now hosted, it will appear at the top of your search.

 

You can now search over 20k data sets on Exversion.

At the end of June we added metadata for over 20,000 data.gov data sets on Exversion, and are in the process of adding thousands of other data sets that are housed on CKAN installations throughout the world.

However, the fundamental problem with CKAN meta data aggregation however is that the CKAN API will not let you query against the actual data set, but instead query what types of data sets there are on each installation of the software, making the platform useless in terms of machine readable data.

This model distributes static .csv files along with secondary and tertiary links to off-site data, making it very difficult to aggregate much of anything aside from the link / meta data.  As such, we’re asking you, the crowd, to help populate these data sets. In order to foster this process we’ve provided a simple applet that will allow you to upload a specific data set.

In the above example, you see that I’ve searched for Crash Statistics by state on Exversion, but the data set is yet to have been imported, namely it needs to be cleaned. Following our style guide, I quickly made the dataset “Exversion ready” i.e. a CSV file, and uploaded to the site with a description of the changes, making the dataset is then available here.

While this is not the perfect solution to the larger problem of not having easily accessible machine readable data, it allows the data community to come together and help make data that has been previously inaccessible, machine readable.

At the same time, while this is the status quo for data housed / linked to on CKAN installations, we’re working on a few projects that should wholly integrate data housed on other platforms.

If you guys have any questions ask them in the comments of feel free to write us at info @ exversion.com

Developers, developers, developers — and the first Exversion Ruby Library

While the video of an excited Steve Balmer shouting “Developers, Developers, Developers” has gone down in history of MSFT and software as a somewhat interesting moment, the man was right in that developers, simply put, rock.

As developers ourselves, we set out building Exversion on the Startup Bus without any plans for it outside of solving our own incessant pain points — those of easily managing open, remapped, and transformed data for use in our own applications — but the response from the developer community at SXSW and later on in NYC and SF made us acknowledge that we may in fact be onto something.

And since launching the closed alpha a bit over two weeks ago, a couple of really cool things happened. The first came in the form of our very own community-built ruby library for accessing the Exversion API. It was built by @bswinnerton and you can grab it here on GitHub. Thanks a million, Brooks!

The Sad State of Open Data

While the wealth of applications built on top of open data is ever increasing, a fundamental problem persists in  surfing open data websites. In short you could say that the entire process is nothing short of  a sad affair.

For example, take a complete listing of every farmer’s market in the United States, that includes the types of produce one can typically find there, or demographics on the Martial Status of Active Military Personnel. The variety is simply astounding, and while these are only two examples, there is just so much data out there. However getting to those bits of good data can be quite a pain.

People who love data see how gleaming insight from even static sets can help us hack everything from public policy to our daily commute. Unfortunately in order for developers to build these tools it’s not enough for data to be open and public, it also has to be accessible and frequently it isn’t.

Welcoming you to Exversion from a highway in Georgia.

Headed for SXSW we left New York on Sunday at approximately 8 am. Since then, we’ve seen the White House, stopped in Raleigh to catch the back end of a Hackathon and give a wholly unprepared pitch to a few entrepreneurs and investors after only conceiving the idea a few early hours before.

What is this madness you may think? Easy. It’s Startup Bus and we’re Exversion, a new startup founded on the road that’s launching the world’s first ever social collaboration platform for data.

What’s this mean? Well for the many of us who work with data, and more frequently than not, open data, we all run into the same problem. Spend hours diving in through countless resources, only to download a dataset that fits the project you want to work on, but find that it’s either totally unusable, or, that you’ll spend the equivalent amount of time in excel or google refine getting the data to do what you actually want to convey.