To cache or not to cache

A colleague recently asked us about offering some advice on delivering a fast web service  to a client who was using a combination of ArcGIS for Desktop and the web. The datasets are from the Ordnance Survey’s OpenData offering.

 

 

The datasets included:

1. OS Street View.

2. OS Vector Map District.

3. OS 50k  Raster.

4. OS 250k Raster.

5. OS Miniscale.

 

Their challenge was working out whether to deliver a cached map service to a client, a dynamic map service or a mixture of a cached map service and the original data as a Geodatabase? The issues he needed to deal with included the time it took to create the map caches versus performance versus utility versus cost. A multi-layered issue but ultimately it boiled down to answering the question ‘to cache or not to cache?’

Actually, his question was answered initially by another question, why bother? Consider signing up with Esri UK’s free cloud-enabled map services and become productive immediately. Leave the caching work to us. This is a route that a lot of organisations are already doing and Esri UK has been happily serving up fast, visually pleasing map services since the service went live in late 2011. However, the client in question was ‘air-gapped’ and could not connect to the internet for security reasons.

So back to the original question. Our recommendation is that it’s often best to cache your data rather than serve it out dynamically.

Server Performance

The resultant CPU and RAM for serving a cached map is tiny compared to the very CPU and RAM intensive operations of rendering the same image dynamically. Most applications will be unable to scale up to heavier use if the base map is not cached.

A map service that is cached can therefore scale very easily as more and more users are added.  Esri UK’s cloud-enabled map services are happily serving a vast number of map views a month and a high level of concurrent users with little effort.

A dynamic service would be unable to get close to the performance of a cached map service as users increase. Parity between dynamic and cached map services exists only at very low levels of usage, say one user. Also, LocalView Fusion[1] templates are designed for cached base maps and some functionality is limited when a dynamic base map is used.

Architecture

There are also architectural differences when using a cached map service:

 o   A web browser can cache tiles it has requested previously, thus reducing the number of hits each client makes on the services over time. Similarly, most users may have some sort of cache or proxy server that also performs the same task but at the network level.

 o   Web servers are designed and optimised for serving lots of files off disk and/or memory; it is how web apps should work.

 o   By comparison dynamic images usually involve a number of resource intensive tasks including: data search, results retrieval, image generation, disk I/O to save the image to disk (even when streaming as MIME[2]) and other such steps.

Other Considerations

Of course, a cached map service isn’t a magic bullet to all of one’s requirements. There are a number of issues to think about: 

  1. Time: One needs to spend time (in some cases, a lot of time) to produce the map cache. The process is dependent on many factors: the number of scales, image type and resolution to pick out three. With the need to potential roll out updates quickly, cached map services may not be the most agile option.[3]
  2. Size on disk: Disk space usage will be a concern as a map cache can take up a significant amount of space. The Esri UK map services currently have a disk foot-print of 3 terabytes (and growing) – all this has a cost in storing and managing.  
  3. Caching Processes: Resources required for creating cache – multi-servers may be required to reduce time but it increases cost. Faster servers may need to be provisioned.
  4. Update Frequency: If the data changes a lot, then it potentially means there is a need to re-cache the data; either all of it the changed portions. Workflows need to be developed to enable this introducing an overhead.
  5. Less Dynamic: If a client needs to perform client side processing such as online digitising, dynamic rendering of layers or some other function; a map cache may not be suitable.
  6. Scale limits: The largest available scale of a map cache may not contain enough detail for certain workflows and organisations.

So in some cases a more subtle and nuanced workflow may be needed such as combining the advantage of both dynamic and cached map services. For example, a client may require data to be available down to 1:500, 1:250 or 1:100 which is just not practical to build map caches using current prices and resources.  So a viable solution would be to use vector data from a File Geodatabase as a base map for these scales, then utilising cached raster maps as the base map once you’ve zoomed out far enough.

For those interested you can see the cache levels and hardware specifications we typically use for building national caches here.

 


[1] Esri UK’s Platform-as-a-service offering

[2] http://en.wikipedia.org/wiki/MIME

[3] Though many options and alternative workflows exist to maintain a continuously changing map cache; we will explore some of these options in future blog posts.

[4] http://aws.amazon.com/ec2/instance-types/

Tip: Set your region in ArcGIS.com for UK featured content

Earlier this month it was announced that there had been some updates to ArcGIS Online. I wanted to highlight one of these enhancements in particular which I believe will be of interest to UK based users.

It is now possible to localise ArcGIS Online for the UK. Defining your region sets the featured maps on the home page, content in the gallery, and the default extent of new maps in the ArcGIS.com map viewer to UK specific content.

The carousel on the ArcGIS.com home page will be populated with UK specific content as shown in the image below. Also, if you navigate to the Gallery then you’ll see that this also now contains UK Maps, Web and Mobile Apps.

Before seeing these changes you will need to modify your region in your ArcGIS.com profile. To do this follow these steps

  • Ensure that you’re logged into ArcGIS.com
  • Click on your name in the top banner to access your profile
  • In the Region options select “United Kingdom” (see below)
  • Click the Save button

Whilst you’re on your profile page, why not ensure that you’ve included some information about yourself and perhaps upload a photo of yourself?

The other major benefit to setting your region is that all new webmaps that you create with the ArcGIS.com map viewer or ArcGIS Explorer Online will initialise with the extent of the UK. This means that you wont have to zoom and pan to the region before starting to create your map.

The maps and other content that appear in the UK Featured Maps and the Gallery will continue to be refreshed so keep an eye out for new content. Also, if you’ve created a map that you wish to be included in these sections then please let us know. 

GIS, Crowdsourcing, Social media and Crisis Mapping, Day 2

The second day started early and switched over to what the conference called 'self-organised' talks where those who have something to say can put up a title and summary of their talk and see who would sign up for them. Last night, a whole bunch of topics were put forward for consideration and throughout the night, delegates were voting. The more popular the talk, the more people will sign up for them. The boring ones got zilch. Quite democratic in my view and kept up with the very self-help, 'doaucracy' nature of this conference. The top ten (or so) choices were then split across 6 different rooms across five time slots. Yes, I can count - some talks were repeated while empty slots were used for social chat and networking. Additionally, there was a technology demonstration area in the main foyer of the conference hall with people milling around.

I attended a number of these self-organised talks:

'Making Crisis Mapping data information actionable for the humanitarian community.'


'Best practises for verifying crowdsourced data'


'Google Mapping Tools. What's new? Q&A and Brainstorming'


I don't want to go through a complete regurgitation of the above talks (they are available on the Crisis Mappers site) - but I can summarise some as follows:

Making crowdsourced data, 'actionable' remains a problem. While everyone is familiar with twitter, facebook etc. Representatives of 'user' organisations, especially the major UN agencies such as the World Food Programme (WFP), UNICEF etc, say it would be very unlikely that they would rely solo on crowdsourced data to trigger relief actions. Certainly crowdsourced data would give a strong corroboration to more established lines of communications and could speed up the pace of actions. For example, tweets (if they're geocoded) could give strong evidence of a specific event with a geographical property. Very useful for speedily allocating resources for example. However, it is unlikely that an international response will be triggered off by a single tweet or a million tweets. There are major issues around the standard of data collected via crowdsourcing. Data collected gives an indication of something but not the priorities. The utility of crowdsourced data as an exploratory tool was readily acknowledged.

Conclusions

  • We must be accountable for data we are gathering and taking action on it.
  • The use of social media might change the way humanitarian agencies work by making them accountable to the beneficiaries through rapid feedback. Once you increase understanding and knowledge within the beneficiaries, and increase the quality of information, you increase their ability to articulate how they evaluate the impact of humanitarian agencies.
  • The goal of a map is not the map itself; it’s the data behind it. What created it and how it is represented. Maybe we need to look into sorting the data differently?


Verifying crowdsourced data. Can one trust crowdsourced data? A great example of trusted crowdsourced data is OpenStreetMap (OSM) - where tens of thousands of users each can edit any part of the OSM map with their work being peer reviewed, probably in real time. However, the thrust in the talk was the more well known crowdsourced data sources which are very subjective in nature. Examples include tweets, short wave radio and SMS to name three. Questions over quality, accuracy and 'trustfulness' were all explored. One cannot verify and QA a tweet, can we? Thus the problem of event-driven data, ephemeral in nature; very fluid and subjective in content poses significantly problems to decision makers. However, if this data is geolocated (many can be), if there's 1000s of tweets from one area, if all are about the same event (told through the eyes of the tweeter) - surely this is significant?

Conclusions:

Verifying data from unknown sources, essentially collecting data from those who shout loudest or longest does fill most GIS professionals with dread. It isn't empirical enough. In many cases, the inability to verify the data, in terms of accuracy, known standards and comprehensiveness isn't an issue. The data occurred and the value of the data is that it is being generated and therefore an indication of some event in a specific time and place and therefore worthy of inclusion as part of the overall decision making.

My own thoughts:

The entire event was a success with all participants hungry for their work to be mainstreamed and accepted into the humanitarian practise. GIS and intelligent mapping was evident throughout the conference with everyone aware of the utility of having a location or place to link up a wide variety of ephemeral crowdsourced data. Data and information remain big topics as everyone tries to grapple with the well known issues of access rights, ownership, data standards, interoperability, quality and in the case of social media, trust.

The International Network of Crisis Mappers is the largest and most active international community of experts, practitioners, policymakers, technologists, researchers, journalists, scholars, hackers and skilled volunteers engaged at the intersection between humanitarian crises, technology and crisis mapping. The Crisis Mappers Network was launched by 100 Crisis Mappers at the first International Conference on Crisis Mapping in 2009. This website has since been accessed from 191 different countries. As the world's premier crisis mapping hub, the Network catalyses communication and collaboration between and among crisis mappers with the purpose of advancing the study and application of crisis mapping worldwide.


The purpose of ICCM 2011 Geneva was to bring together the most engaged practitioners, scholars, software developers and policymakers at the cutting edge of crisis mapping to address and assess the role of crisis mapping and humanitarian technology in crisis response. Following the myriad of responses to the Haiti 2010 earthquake, the crisis mapper community is fanning out into new domains. A number of reports highlight a great deal of excitement over the potential of Information and Communications Technologies, new social media as well as crowd-sourcing to strengthen planning and delivery of aid.

Oh, Willow from Geeks without Borders took notes as on her iPad and to be honest, her notes are far, far better than mine!

Enjoy!

GIS, Crowdsourcing, Social media and Crisis Mapping, Day One.

The purpose of ICCM 2011 Geneva is to bring together the most engaged practitioners, scholars, software developers and policymakers at the cutting edge of crisis mapping to address and assess the role of crisis mapping and humanitarian technology in crisis response. Following the myriad of responses to the Haiti 2010 earthquake, the crisis mapper community is fanning out into new domains. A number of reports highlight a great deal of excitement over the potential of Information and Communications Technologies, new and social media as well as crowd-sourcing to strengthen planning and delivery of aid.
Read More

LocalView Fusion Service Pack 5

After 2 solid months of development, LocalView Fusion Service Pack 5 is in the final stages of testing and will be available for download next week (customers will get an email containing the download details). The release includes two completely new GeoTemplates plus a list of product improvements. This post describes a few of the headline features that will be in SP5.

New Embeddable Map GeoTemplate

Customers with the Community bundle will get the new Embeddable Map GeoTemplate. This provides a simple way to create and embed GIS viewers into your website using very few lines of HTML and CSS, reducing development time without sacrificing functionality. Plus you don't need to have any experience in Esri programming to add an Identify tool, address search or legend - this can all be set up in the provisioning site.  Here’s what it looks like in action (screenshot taken from our main Esri UK website):

 

New ArcGIS Viewer for Flex GeoTemplate

This GeoTemplate will be available if you have the Business bundle of LocalView Fusion. It is a modified version of Esri’s sample Flex application which we have integrated into the LocalView Fusion platform. So in addition to the rich functionality already available in the original application (advanced attribute searching, editing, time-slider, etc), we have included our UK address searching functionality, printing suite and full legend/layer control. You can get more information about the standard Flex viewer here if you’re eager to find out more about what it can do.

Improvements and bug fixes

As well as the new GeoTemplates we have been listening to your feedback and have added a range of new options, services and fixes to the system. Here is a sample of some of them;

  1. Multiple basemaps (Web Publisher, Navigator, Embeddable Map and AGS Viewer for Flex).
  2. Web Publisher: Property level reverse geocoding.
  3. Enhanced support for WMS services.
  4. The ability to print from secured map services.
  5. Testing environments extended to include new browsers (Chrome and IE9) and server environments extended to include Oracle 11g.
  6. Survey actions updated to use the GeoProcessing content type.

So look forward to SP5 and check back here for more updates about LocalView Fusion in the next few weeks.