Data, Data Everywhere

When I started this project, data was scarce. FOIA laws were new and local government didn't think much of them. I was fortunate to have located a market where I could get a regularly updated dataset to use for development.

On a computer named "hydra" I'm running a data processing monster. It's been rewritten 5 times before. It takes datasets of any form, from anywhere, requiring any convoluted procedure to acquire and manages the complex task of converting them into a continuously freshened coverage areas for the market analysis engine.

I'm making calls and sending emails. "Assessor's office, please."  I'm looking for more coverage areas, new ways to feed the hydra.

And the news is good. Local government is now used to requests for data. There are still a few that charge prohibitively high FOIA fees, but there's less of them now. The responses I'm getting are encouraging.

The hydra is purring. The latest new coverage area got integrated in under two days. The first one took months.  Another few markets under my belt and I'll get it down to two hours.

When I first talked about the idea that the hydra is built on, I think my exact words were: "Sure we could build one, if we had infinitely large piles of time and money.  Failing that, I'd say it's impossible."

:grin: