Surveying, Mapping and GIS

Exploring all aspects of mapping and geography, from field data collection, to mapping and analysis, to integration, applications development and enterprise architecture...

  • Geospatial Technology, End to End...

    Exploring all aspects of mapping and geography, from field data collection, to mapping and analysis, to integration, applications development, enterprise architecture and policy

Think Globally, Code Locally

Posted by Dave Smith On 5/23/2010 10:35:00 AM 5 comments

One of those key philosophical notions that circulates among many stewardship and advocacy groups is to think globally, act locally - The same notion can be applied in best practices in solutions architecture, requirements development, and coding principles, where it applies to so many things - in terms of true enterprise architecture and integration approaches, in terms of data stewardship, and for public sector agencies, in terms of transparency and open government.

Each and every day, folks craft queries for accessing complex data, they write business logic for analysis and so on - and more often than not, they then wrap it up with a nice glossy user interface.  MVC and many other best practices applied, check...  But here is where another key opportunity to really open things up is lost.

Let's take the example of state or federal agencies - many of them are doing great things, in terms of publishing datasets (Open Government, Geospatial One-Stop, State Cartographers' Offices and Clearinghouses, and so on).  They are doing great things in terms of reporting tools, web queries and lookups, and mapping and visualization tools.  But...  As wonderful as all of these efforts are, imagine how powerful it would be if these could be linked and integrated, from agency to agency?  Imagine, for example, applications pulling in data from NOAA, NASA, EPA and other agencies?  Imagine state environmental agencies pulling in information from federal environmental agencies - the power of open government and transparency begins to multiply and become more and more powerful by orders of magnitude.

So what are we talking about?  Web services and APIs.  Not even anything complex, it could be as simple as REST and JSON, as in the space of public-facing applications, for a vast majority of data-centric or analytical applications, we are generally talking about read-only, non-transactional, nonsensitive data.  Why web services or APIs?   Isn't something like publishing datasets to enough?  Publishing datasets to someplace like is fine - but consider this - if someone downloads your data and integrates it into their own standalone application, what control do you then have over it?  Precious little.  When you refresh your database afterward and post a new dataset, is there any assurance that they will come back and get a fresh copy and keep up to date, or do you begin to incur a liability of your data being out of sync as it is represented in someone else's public-facing application?  Absolutely.

If, on the other hand, you post an easy-to-integrate RESTful service or API, you make it far easier to have your data current.

Criticisms?  Yes, there are many, but we can look at those as well - some might push back and say, "is there sufficient bandwidth and infrastructure for hosting those web services?"  The mere fact of web services in and of themselves generally incur no more new bandwidth or infrastructure demand than the public-facing web applications that you are already building.  So if that's really a concern, then maybe you shouldn't be building ANY applications.  The argument is generally a red herring.

Security concerns?  Same deal - you are already exposing the data and business logic via your web applications - if you think the web service poses new security concerns, then perhaps you haven't adequately thought through your EXISTING security.  So another red herring.

"They will come and take all our data!" - Again, why would they, if they know they can grab it dynamically and that you will be assured of their reliance on them by continually posting data refreshes?

What about poorly-behaved queries against your web service that might bring your database to its' knees?  Same thing - those are things that really should be handled in the logic for your existing application.  Better yet, think ahead a bit more and provide options - query filters, such as by a data attribute or domain, bounding boxes for geodata, and so on.

So - in conclusion, this type of thinking can up developers' approaches to how they deal with security, queries and filters, and so on, and generally tighten things up - but in the meantime, also provide a tremendous opportunity for data sharing and integration, whether within an agency, company or office, fostering collaboration across departments and programs, or outside of one's own organization, fostering collaboration across agencies and levels of government, across industry, academia and so on.

An individual organization can try all they like to anticipate all of the public's thoughts, needs, interests, they can try to walk in their stakeholders' footsteps - but ultimately, one can never fully anticipate all possibilities, and there are a lot of bright people out there with many untapped ideas - so why not open it up and facilitate that larger collective brainstorm of ideas by providing services ready to integrate?  Let's take things like the Open Government Initiative and other good things ongoing in government and elsewhere to the next level - dynamic, services and resources oriented architecture...

5 Response for the " Think Globally, Code Locally "

  1. Dave,

    Why don't more people use existing models for data exchange like NIEM and UCore? I understand that they are currently focused towards IP and Intel but both frameworks appear to have ways to extend their current architecture to accommodate the need for different types of data.

    I am by no means an expert when it comes to either of these models but I do see the value in implementing such technologies. For example, if the USDA and EPA had to share data, they could simply query and migrate data from one model to another. NIEM or UCore could, in essence, broker the exchange...

    Data modeling should not be limited to deep-thinking hippies sitting around on bean bag chairs working on how to best tie our data together...we should all be thinking about it and while web services are great and do promote exchange and sharing, they still just add an abstraction layers that needs to be dealt with. Thoughts?


  2. Anonymous says:

    In the environmental world, States & Tribes already use the NEIEN web services and Exchange Network to share data back & forth with each other and with the EPA. This network of web services has been in place for quite a few years and is working well. We have invested too much in the Exchange Network over the last decade., so why in the world would we want to scrap it and start using something designed for and controlled by the law enforcement and homeland security types? If that community wants to access and use our environmental data, they can easily develop a bridge or connector between their system of web services and the Exchange Network web services - and vice versa.

  3. Anon,

    Very good point and actually, this is the kind of information I was looking for. To be honest, I didn't know that NEIEN was that widely used. But, that's probably because of my lack of knowledge of what it is/does.

  4. The hope is that the models are not single-purpose; that they instead would be extensible... It would be interesting to see crosswalks between NEIEN, UCore, NIEM and others - there is likely much commonality where it comes to certain elements.

    Some good proofs-of-concept would be to build apps that allow flows across all of these...

  5. I visited your website, the information which you have shared in your site is very much standard and presentation of pages is very legible. Just would like to say thank you for the information you have shared. Just keep on writing this sort of post. I will be your devoted reader. Thanks again.
    If you looking for marketing service then geomarketing can help you.