Surveying, Mapping and GIS

One of those key philosophical notions that circulates among many stewardship and advocacy groups is to think globally, act locally - The same notion can be applied in best practices in solutions architecture, requirements development, and coding principles, where it applies to so many things - in terms of true enterprise architecture and integration approaches, in terms of data stewardship, and for public sector agencies, in terms of transparency and open government.

Each and every day, folks craft queries for accessing complex data, they write business logic for analysis and so on - and more often than not, they then wrap it up with a nice glossy user interface.  MVC and many other best practices applied, check...  But here is where another key opportunity to really open things up is lost.

Let's take the example of state or federal agencies - many of them are doing great things, in terms of publishing datasets (Open Government Intitiative, Data.gov, Geospatial One-Stop, State Cartographers' Offices and Clearinghouses, and so on).  They are doing great things in terms of reporting tools, web queries and lookups, and mapping and visualization tools.  But...  As wonderful as all of these efforts are, imagine how powerful it would be if these could be linked and integrated, from agency to agency?  Imagine, for example, applications pulling in data from NOAA, NASA, EPA and other agencies?  Imagine state environmental agencies pulling in information from federal environmental agencies - the power of open government and transparency begins to multiply and become more and more powerful by orders of magnitude.

So what are we talking about?  Web services and APIs.  Not even anything complex, it could be as simple as REST and JSON, as in the space of public-facing applications, for a vast majority of data-centric or analytical applications, we are generally talking about read-only, non-transactional, nonsensitive data.  Why web services or APIs?   Isn't something like publishing datasets to Data.gov enough?  Publishing datasets to someplace like Data.gov is fine - but consider this - if someone downloads your data and integrates it into their own standalone application, what control do you then have over it?  Precious little.  When you refresh your database afterward and post a new dataset, is there any assurance that they will come back and get a fresh copy and keep up to date, or do you begin to incur a liability of your data being out of sync as it is represented in someone else's public-facing application?  Absolutely.

If, on the other hand, you post an easy-to-integrate RESTful service or API, you make it far easier to have your data current.

Criticisms?  Yes, there are many, but we can look at those as well - some might push back and say, "is there sufficient bandwidth and infrastructure for hosting those web services?"  The mere fact of web services in and of themselves generally incur no more new bandwidth or infrastructure demand than the public-facing web applications that you are already building.  So if that's really a concern, then maybe you shouldn't be building ANY applications.  The argument is generally a red herring.

Security concerns?  Same deal - you are already exposing the data and business logic via your web applications - if you think the web service poses new security concerns, then perhaps you haven't adequately thought through your EXISTING security.  So another red herring.

"They will come and take all our data!" - Again, why would they, if they know they can grab it dynamically and that you will be assured of their reliance on them by continually posting data refreshes?

What about poorly-behaved queries against your web service that might bring your database to its' knees?  Same thing - those are things that really should be handled in the logic for your existing application.  Better yet, think ahead a bit more and provide options - query filters, such as by a data attribute or domain, bounding boxes for geodata, and so on.

So - in conclusion, this type of thinking can up developers' approaches to how they deal with security, queries and filters, and so on, and generally tighten things up - but in the meantime, also provide a tremendous opportunity for data sharing and integration, whether within an agency, company or office, fostering collaboration across departments and programs, or outside of one's own organization, fostering collaboration across agencies and levels of government, across industry, academia and so on.

An individual organization can try all they like to anticipate all of the public's thoughts, needs, interests, they can try to walk in their stakeholders' footsteps - but ultimately, one can never fully anticipate all possibilities, and there are a lot of bright people out there with many untapped ideas - so why not open it up and facilitate that larger collective brainstorm of ideas by providing services ready to integrate?  Let's take things like the Open Government Initiative and other good things ongoing in government and elsewhere to the next level - dynamic, services and resources oriented architecture...