Here come ArcGIS 10.4 and ArcGIS Pro 1.2

ArcGIS 10.4 will be released next week, in fact the help pages are live already. Both ArcGIS Server and ArcMap continue to evolve to reflect the needs of the user communities. If your organisation is running Portal alongside ArcGIS Server there are a raft of changes in the new release. A new version of ArcGIS Pro will follow on from the 10.4 release, so here's an overview for ArcGIS 10.4 and ArcGIS Pro 1.2.

Read More

Some hidden gems in ArcGIS 10.2.1

The ArcGIS 10.2.1 release last month was in part a stability release but also included some interesting new features. One of the most significant was the new ArcGIS for Server capability to Enable feature services for offline map use. ArcGIS for Desktop saw new capabilities added, mostly to the Geoprocessing toolboxes.
Read More

Why are my OGC® Web Services displaced by about 120 metres?

You add your British National Grid data to a new Map Document in ArcMap and publish it to ArcGIS Server, ticking the Web Map Service (WMS) box, getting no errors and all is well; you even add the WMS to another Map Document  and everything matches perfectly.  Later when the user - you know the one who always finds things which go wrong - adds the WMS to their mash-up they call you and inform you that the map is shifted about 120 metres.  Sound familiar?

So you curse, kick the server (not literally, I hope) and get on the phone to Support to complain about a bug in the software.  Unfortunately the bug is not in the software, the omission is in the OGC® WMS Specification and potentially affects all WMS, WFS, WCS (in fact any W*S) Services. 

The user who reported the problem is not using British National Grid (BNG) as the Coordinate Reference System (CRS) of their map but something based on a different datum to GCS_OSGB_1936, which BNG is based upon, such as GCS_WGS_1984 or a projection based on a different datum such as WGS 1984 Web Mercator Auxiliary Sphere.  And the reason why the map is shifted is because no Geodetic Datum Transformation (GDT) has been applied to the published Map Document (see this blog entry for information about transformations).  But why would you expect to apply a GDT if you are only using BNG? 

We will skip the science bit about the W*S Specification until later but focus now on the workaround to overcome the problem that this omission in the OGC® W*S Specifications causes in the publishing of W*S using ArcGIS.

Workaround

After you have added your British National Grid data to the DataFrame in ArcMap you should change the DataFrame CRS to GCS_WGS_1984 and apply a transformation (Choose the transformation as per our earlier blog Understanding which Transformation to choose in ArcGIS)

After adding the transformation change the DataFrame back to British National Grid and save the Map Document, then publish to ArcGIS Server as normal.  If you want to add INSPIRE CRS capabilities to the WMS then a further 2 transformations would need to be added - as explained in the Science Bit.

Your problem is resolved!

The Science Bit – For those of you who wish to know.

The OGC® have been informed of the omission of Geodetic Datum Transformations in Web Services (Change Request 09-187 dated 22 December 2009 – Status Pending) but little has been discussed outside of OGC®.

If the native CRS of the data is based on any datum other than GCS_WGS_1984 then there should always be a GDT between that CRS and GCS_WGS_1984; this is required for accurate calculation of <EX_GeographicBoundingBox> attributes if nothing else. 

The formula to calculate the minimum number of GDT required for a WMS (assuming that each GDT is reversible) is:

where x is the different number of Datum required.

So let’s say you have British National Grid data and you need to show the WMS in Web Mercator and INSPIRE, then you have 3 datums and looking at the formula you require 3 GDT such as:

CRS Required in WMS

GDT

EPSG Code

British National Grid

GCS_WGS_1984

GCS_ETRS_1989

OSGB_1936_To_WGS_1984_7

OSGB_1936_To_ETRS_1989_1

ETRS_1989_To_WGS_1984

5339 *

5338 *

1149

 

*  Look at the blog entry OSTN02 supported in ArcGIS desktop to include these transformations in ArcGIS 10.0.

The following is a graph of the formula.

You have already been shown how to add a transformation between British National Grid and GCS_WGS_1984; to add the transformation between British National Grid and GCS_ETRS_1989 you need to have installed the OSTN02 in NTv2 and then change the DataFrame CRS to GCS_ETRS_1989 and apply the “only” transformation available.  To add the final GDT you need to add some GCS_WGS_1984 data while the DataFrame is in GCS_ETRS_1989 then add a transformation selecting GCS_WGS_1984 and the transformation to GCS_ETRS_1989.

Remove the added GCS_WGS_1984 data, change the DataFrame back to British National Grid, save the MXD and publish to ArcGIS Server as normal.

Jim Sibbald is looking at writing an Add-In for ArcGIS 10.0 to provide a simpler interface to the workaround.

Summary

When preparing W*S services with data other than that based on GCS_WGS_1984 you must always ensure that suitable Geodetic Datum Transformation(s) are added to the Map Document before publishing to ensure that you do not get the user problem displayed above.

Jim Sibbald

 

LocalView Fusion Service Pack 5

After 2 solid months of development, LocalView Fusion Service Pack 5 is in the final stages of testing and will be available for download next week (customers will get an email containing the download details). The release includes two completely new GeoTemplates plus a list of product improvements. This post describes a few of the headline features that will be in SP5.

New Embeddable Map GeoTemplate

Customers with the Community bundle will get the new Embeddable Map GeoTemplate. This provides a simple way to create and embed GIS viewers into your website using very few lines of HTML and CSS, reducing development time without sacrificing functionality. Plus you don't need to have any experience in Esri programming to add an Identify tool, address search or legend - this can all be set up in the provisioning site.  Here’s what it looks like in action (screenshot taken from our main Esri UK website):

 

New ArcGIS Viewer for Flex GeoTemplate

This GeoTemplate will be available if you have the Business bundle of LocalView Fusion. It is a modified version of Esri’s sample Flex application which we have integrated into the LocalView Fusion platform. So in addition to the rich functionality already available in the original application (advanced attribute searching, editing, time-slider, etc), we have included our UK address searching functionality, printing suite and full legend/layer control. You can get more information about the standard Flex viewer here if you’re eager to find out more about what it can do.

Improvements and bug fixes

As well as the new GeoTemplates we have been listening to your feedback and have added a range of new options, services and fixes to the system. Here is a sample of some of them;

  1. Multiple basemaps (Web Publisher, Navigator, Embeddable Map and AGS Viewer for Flex).
  2. Web Publisher: Property level reverse geocoding.
  3. Enhanced support for WMS services.
  4. The ability to print from secured map services.
  5. Testing environments extended to include new browsers (Chrome and IE9) and server environments extended to include Oracle 11g.
  6. Survey actions updated to use the GeoProcessing content type.

So look forward to SP5 and check back here for more updates about LocalView Fusion in the next few weeks.

Debugging GP Services

So here’s the deal: you’ve spent ages perfecting a GP model, incorporating loads of different tools and adding custom touches in python. In fact, maybe you’ve even gone the whole hog and scripted your GP tool from scratch – and it works like a charm in ArcMap. Bingo.

You publish it to ArcGIS Server, you pull up the ArcGIS Rest Services Directory and navigate through to the GP Server Rest Endpoint. You painstakingly harvest some valid JSON to define your input parameters, and patiently wade through the ‘Invalid value for parameter X’ messages. You hit ‘Submit Job’…..and it goes kaput. Nothing, zilch, nada, zippo.

Except of course:

 

Where do you go now? Why does a perfectly valid GP tool break the minute you turn it into a GP Service? Well, sometimes these things happen.

First things first, I extracted the whole thing to one big Python script, and I began to pepper it with arcpy.AddMessage() calls, mainly logging vital information like, err,  arcpy.AddMessage(“I am here”) and arcpy.AddMessage(“Now I’m here”). I saved the script, refreshed the model in ArcMap, saved the MXD, refreshed the map/GP services, cleared the Rest cache, (did a handstand and shut my eyes), and…

 

Hurrah! No, wait…

So I took a different approach. And I’m delighted to say that after a lot of huffing, puffing and flapping; a few cups of tea, the odd lunch break; maybe a quick chat about the weather with someone passing by, a few more cups of tea, a comfort break; some cheese-on-toast, a quick distraction to fix a LocatorHub web service, a stroll into town, and back again….

 

Um. Yeah. Cheese on toast never fixed anything.

So I bit the bullet and decided to recreate my script, line by line, to pin down the problem. With a deep breath, I started out with just a simple opener:

 

Failed

Soon I was pegged back to this:

 

Failed

What is going on?

My colleague Simon suggested I take a look at some log files in C:\Program Files (x86)\ArcGIS\Server10.0\server\user\log\. That was a good start. In one of them I came across the following (emphasis added):

<Msg time='2011-08-12T15:07:22' type='ERROR' code='20010' target='pipelines2.GPServerSync' methodName='GPServerSync.CheckMessages' machine='TAYLOR' process='3156' thread='6244'>Executing (GetIntersections2Model): Model1 Start Time: Fri Aug 12 15:07:22 2011 Executing (GetIntersections2): GetIntersections2 Start Time: Fri Aug 12 15:07:22 2011 ERROR 000576: Script associated with this tool does not exist. Failed to execute (GetIntersections2). Failed at Fri Aug 12 15:07:22 2011 (Elapsed Time: 0.00 seconds) Failed to execute (GetIntersections2Model). Failed at Fri Aug 12 15:07:22 2011 (Elapsed Time: 0.00 seconds)</Msg>

Then it dawned on me. While I had been carefully storing my MXDs and data in a location visible to the ArcGIS SOC process (critical for ArcGIS Server permissions) I had stuffed my python script in a folder inside my user space. ArcGIS Server doesn’t have permissions to see in there. Such a simple and obvious mistake. But such a time-waster!

Having moved the script outside my user space to somewhere more sensible, I regenerated my model, republished my MXD and once again, cleared my Rest cache, and my script succeeded first time.

 

But hang on a minute. Where are my debug messages?

Remember, I had bunged a load of arcpy.AddMessage() calls into my script just to try and get some debug info. Hey – I’d even got so frustrated I used an arcpy.AddWarning() just because I like to live dangerously. But now my script was clearly executing, where was my debug info?

Well, ladies and gentlemen, here’s the crux of it. Your GP service properties window has an option in the Parameters tab, called Show Messages, but its default state is unchecked. You need to turn this on in order to see any of your custom message information (and you’ll need to do this while your service is stopped).

For security reasons, debug information is not shown by default. But providing you’re aware of the risks of exposing potentially exploitable information (eg file names/paths/usernames/etc) you can turn this feature on. That way your end user sees status information, warnings and – most importantly – error messages, which could actually be incredibly useful to them. (After all, as an end user I would want to know that a call to a service has failed.) Plus as a developer I can then give users progress updates via status messages, especially for tasks that may take more than a few seconds to execute.

So in my book, the Show Messages option should  be checked as long as we’re aware of the sensitivity of outputting certain types of debug/status information.

 

(Aside: it is also possible to achieve the same results by changing the ShowMessages property in your service’s CFG file, but you’ll need to restart your whole SOM before the changes will take effect.)

So after all that, I finally had my messages coming through:

 

 

In summary…

If you are having problems debugging ArcGIS Server GP Services

1)      Make sure your GP tool works correctly in ArcMap before publishing it as a service

2)      Make sure all resources for your service – MXDs, toolboxes, data, script files, etc – are stored in a location that ArcGIS Server can access (ie outside your user space)

3)      Check your log files in C:\Program Files (x86)\ArcGIS\Server10.0\server\user\log\ for clues as to why your GP service is failing

4)      Turn on Show Messages and use arcpy.AddMessage() to send debug information to the client.

And if all else fails, ask a colleague called Simon*

*not all colleagues called Simon may have ArcGIS Server, GP and python experience. Use of Simons is at your own risk.

Cloud computing at Esri UK

ISG (Internet Services Group) has for the last six years been managing Esri UK's hosting and web services both for internal and external clients. This was supported through internal and third party hosting suppliers.

The cloud offered a number of differences (I would not use the term 'advantages') over traditional on-premise hosting.

The on-going debate between on-premise and cloud solutions comes down to specific details and requirements within each organisation. The decision to go with one or the other solution should be made only after reviewing each organisation’s unique situation. Cloud computing offers a lot but it should not be regarded as the magic bullet.

ISG started off with an on-premise model but over time migrated towards a cloud-computing infrastructure as provided by Amazon Web Services (AWS) and taking advantage of the utility pricing model where one only pays for what one uses. When a server or resource is not required, it is turned off saving money both in terms of management and power. There is no need to pay for excessive capacity.

AWS had been working with Esri Inc to create ArcGIS Server Amazon Machines Images (AMI) - which are pre-created ArcGIS Server 10.0 virtual machines running on the AWS cloud. ISG have been using these AMIs, editing and reconfiguring them for our specific requirements and commissioning them for use. For ISG, the cloud has meant that managing hardware and software with a small team, isn't such a time consuming task anymore. Gone are the days where one needs to travel down to our hosting centre to change disks. The headache of hardware upgrades are now a thing of the past as AWS continues to add and maintain the underlying hardware. 

For Esri UK the PROS and CONS of a Cloud model of service provisioning include:

PROS:

•Lower Total Cost of Ownership (TCO) - certainly, the cost to the business of setting up a hosted service has dropped dramatically. In the past, even before the first user - a hosted solution would have cost upwards of £1,000s to setup. Now, a few days of work and the service is usually ready.
•Scalable performance to fit business’s needs. If a service has seasonal peaks (i.e. an online conveyancing service or a charity web site) then the ability for the resources to expand in terms of size and power as usage demands it becomes a big plus for us. Resources can be added almost instantly. In the past, one had to purchase extra hardware which would become redundant after the initial peak requirement. The cloud grows as big and as quickly as demand requires or conversely, can shrink down to a minimum. Saving in cost all round.
•Reduce upfront costs on hardware and software means that ISG does not need to spend days working up estimates on hardware costs and maintenance payments.
•Higher server availability and recovery. If one server fails, it is automatically picked up on another server assuming a copy of it was made. Usually, this new server can be online within minutes. Even higher availability can be provided if the solution involves clustering and load balancing.
•Utility based. With a cloud solution we will only pay for the amount of server resource we need and pay for them on a monthly basis, like electricity.

Hosted CONS:
•Bandwidth costs could potentially be very high if user numbers prove incorrect from estimates.
•Not all software programs can be moved into the cloud.
•Depending on the size of the client, it may not be a great price difference between cloud and on-premise.
•Security can be a concern. If a service or a client has specific security requirements, it may not be able to move to a cloud-based solution.
•Downtime. One is completely dependent on the Internet and service provider. If the Internet is down, service could be offline. The risk is of failure is there but it is borne by Amazon Web Services[i] rather than ourselves.
•Continuous cost, after the initial pain of capital expenditure - the cost of running an on-premise recedes to a background cost of staff salary and core infrastructure costs (power and light); it becomes an asset that depreciates in value. With the cloud, one can have a variable and continuous cost that may be difficult to predict. This can cause issues with budget forecasts and estimates for the business.
•Even for simple demonstrations, testing etc - one still has to pay. One doesn't have an 'old' machine handy to load up a test environment.

Since there are several PROS and CONS to hosted models and on-premise models, ISG has found a compromise in a hybrid solution.  A hybrid solution for ISG means that some of our resources are still on-premise (a combination of physical, virtual and internal cloud resources) these are used for testing, demonstrations and for specific clients. Our production/live services are now cloud-based giving us the best of both worlds.

A lot of other areas need to be discussed such as the relevant performance of disk I/O between  cloud resources as opposed to a physical on-premise resource. Security is another large area of concern as well as the necessary changes in one's working habits.

I guess that's for another time.

 


[i] http://www.zdnet.com/blog/foremski/the-irony-of-a-cloud-knocking-out-amazons-ec2-cloud-services/545

Creating Augmented Reality POIs from ArcGIS Server data

At the Esri UK Annual User Conference in May, we did a quick Augmented Reality (AR) demo showing how to take an ArcGIS Server point feature service and display the data on a mobile phone. To do this, we used a free AR app called Layar.

AR refers to any technology that lets you see a layer of digital information overlaid on the real world. For example, using AR you can hold your mobile phone's camera in front of you and see public transport stations in that direction, as we did with London Underground stations for our demonstration (see the screenshot below).

Augmented Reality takes advantage of the camera, compass and GPS facilities found on modern smartphones to position the digital data correctly, and AR apps can help you discover a great deal of spatial information about the environment around you.

We published a feature service containing Underground stations near our conference venue in Central London and then wrote a proxy webpage to accept HTTP requests from the Layar client, translate these into ArcGIS Server REST calls, and send back ArcGIS Server point features in the JSON format that Layar expects to receive. Layar then does the hard work of rendering the image on the mobile phone.

It's quite easy to create a proxy webpage between ArcGIS Server and Layar using your favourite development technology. Since we had IIS running on a Windows Server, our Layar/ArcGIS proxy was a single ASP.NET page using the DataContractJsonSerializer class to read and generate JSON objects, but any development technology that can send HTTP requests to ArcGIS Server's REST API and manipulate JSON text will do the job just as nicely.

An example of the JSON output that our proxy page produces is given below

{
    "errorCode": 0,
    "errorString": "ok",
    "hotspots": [{
        "actions": [{
            "activityMessage": "Edit name",
            "activityType": 1,
            "autoTriggerOnly": false,
            "autoTriggerRange": 0,
            "closeBiw": false,
            "contentType": "text\/html",
            "label": "Edit name",
            "method": "GET",
            "params": [],
            "showActivity": true,
            "uri": "http:\/\/example.com\/layar\/editName.aspx?id=18"
        }],
        "distance": 937,
        "id": "station_18",
        "imageURL": null,
        "lat": 51523273,
        "line2": "Warwick Avenue, London, W9 2PT",
        "line3": "Bakerloo",
        "line4": "Zones: 2",
        "lon": -183678,
        "title": "Warwick Avenue",
        "type": 2
    }],
    "layer": "yourlayernamehere",
    "morePages": false,
    "nextPageKey": null
}

One quirk of Layar is that the latitude and longitude coordinates are given as integer microdegrees: standard WGS-84 coordinates multiplied by 1 million (1000000) and then rounded off to a whole number.

For more information on Augmented Reality and GIS, check out this ArcUser article and this video from the 2011 Esri Inc Developer Summit on Augmented Reality using ArcGIS and Layar.

 

Speed up your tiled map services

I want to tell you about a simple trick that will speed up the loading of map tiles from ArcGIS Server tiled map services in all browsers. We’ve recently incorporated this into LocalView Fusion and it requires only a handful of ingredients:

  • A tiled map service running on a server you control.
  • A few lines of JavaScript
  • Some extra "A" records (subdomain names) for the domain name of your site (use your hosts file if you are developing this and wish to test it)

The problem: Blocking

Browsers can only make a limited number of HTTP requests at once. In modern browsers, like Internet Explorer 9, Chrome and Firefox 4, up to 6 HTTP connections can be made per host name (although in FF at least you can configure this). So if your tiled map service is running on a server with the address like http://www.myserver.com, then these browsers can send up to 6 tile requests at once before they have to stop and wait for the responses to come back.

If more than 6 requests are sent to the same host name, the browser will wait for responses to be returned before sending requests 7, 8, 9 and so on. This is called “blocking”.

However, a browser will usually need to display more than 6 tiles at once to show a map. For larger resolution screens there may be 20 or 30 tiles making up the full map display. So after the first 6 tiles have been requested, the browser will “block” until it can send more tile requests.

The solution: multiple subdomains, and some code

Let me first make it clear that this is not a new trick, I just haven’t seen it implemented in an ESRI web API client yet. Google, Bing and Open StreetMap tiles are routinely requested from multiple subdomains in clients and APIs for those services: for example the OSM tile URLs are a.tile.openstreetmap.org, b.tile.openstreetmap.org, c.tile.openstreetmap.org and so on. This allows 6 tiles to be requested from each subdomain: 18 at once. (Older browsers may have a lower concurrent connection limit: IE7 is limited to two per subdomain for example).

This trick can be applied to ArcGIS Server map services as long as you have control over the registering of the extra “A” records for your domain (a.myserver.com, b.myserver.com and so on). The rule of thumb is that 3 or 4 “A” records will speed up requests but beyond this diminishing returns apply due to the extra burden of DNS lookups.

So – step one: register 4 extra “A” records for the domain where your server is running. In our example these would be:

a.myserver.com

b.myserver.com

c.myserver.com

d.myserver.com

The code

My example here is based on the JavaScript API but there is no reason why the same approach wouldn’t work in Silverlight or Flex, as long as you can implement a custom layer type using those APIs. You can do this easily in the JavaScript API as described in this article on the ArcGIS Resource Centre.

It is easy to extend the ArcGISTiledMapServiceLayer class in the ESRI JavaScript API, thanks to the Dojo framework upon which the API is built. By defining a new class that inherits from a base class, and implementing an override for the properties and methods you wish to change, the amount of code required can be kept to a minimum.

To make multiple requests to the same map service, the code will need to do the following:

  •  Read the service metadata from the “normal” map service URL as usual. This is the URL passed into the constructor method of the ArcGISTileMapServiceLayer class.
  • Define new properties for the list of server URLs to use, and to keep count of which one each tile request is using.
  • Override the getTileUrl method, to loop through the list of subdomains in turn and make a request from the next one in the list.

Here is a code sample of the custom layer class as used in LocalView Fusion:

 

dojo.provide("CustomLayer"); //define the namespace and class
dojo.declare("CustomLayer", esri.layers.ArcGISTiledMapServiceLayer, {

    /* Extends the ESRI class by adding a list of servers. 
    /* This needs to be manually populated when 
    *  the object is created.
    */

    servers: [],
    serversLength: -1,
    serverIndex: 0,


    getTileUrl: function (level, row, col) {
        var idx = 0; 
        //check how many URLs there are.
        if (this.serversLength < 0) { 
            this.serversLength = this.servers.length;
        }
      //if there is more than one URL, get the next in the list.
      if (this.serversLength > 0) { 
            // Multiple urls
            idx = this.serverIndex++ % this.serversLength;
            return this.servers[idx] + "/" 
                   + this.url + "/tile/" 
                   + level + "/" 
                   + row + "/" 
                   + col + ".png";
      }
      //if we don’t have extra URLs, just use the default one.
      else { 
            // Original single url
            return this.url + "/tile/" 
                   + level + "/" + row + "/" 
                   + col + ".png";
      }
  }
});

 

 

And here is a sample of how it might be used in an application:

var basemap = new CustomLayer(
 "http://www.myserver.com/ArcGIS/rest/services/MapServiceName/MapServer"
);
basemap.servers = [
 "http://a.myserver.com/ArcGIS/rest/services/MapServiceName/MapServer",
 "http://b.myserver.com/ArcGIS/rest/services/MapServiceName/MapServer",
 "http://c.myserver.com/ArcGIS/rest/services/MapServiceName/MapServer"
];
map.addLayer(basemap); //assuming "map" is the name of our map object.

To conclude

In our testing, using a modern browser like Firefox 4, Internet Explorer 9 or Chrome, the effect is that ArcGIS Server tiled maps load in the browser much faster than before, up to 24 tiles coming in at once compared to the batches of 6 you would normally get with those browsers.

If you use IE7 this will be less impressive as it uses a strict implementation of the HTTP 1.1 spec, and can only make two requests per host - even so, this will fetch 8 tiles at once using this method.

If you're using IE6, well....I don't suppose faster tile loading is going to make much difference as the general HTML rendering and script execution performance is so bad to start with!

This functionality will be in the next release of LocalView Fusion, but if you are running ArcGIS Server with a registered domain name and can make the edits I've described here to your application, then there's nothing stopping you implementing it right now.

Printing in ArcGIS Server

Printing in ArcGIS Server is one of those topics we get asked about on a regular basis. If you are looking for something more than the out of the box functionality the solution was usually to roll up your sleeves and get stuck into ArcObjects. At 10 however Esri released the arcpy.mapping Python module. This gives a relatively simple option for some nice prints. If you haven't seen it before Esri have written a great getting started blog article with some nice examples. Well worth a read:

http://blogs.esri.com/Dev/blogs/arcgisserver/archive/2011/04/12/An-introduction-to-arcpy.mapping-for-ArcGIS-Server-developers.aspx

Enjoy!