Vector Tiles Part 1 - Creation

Vector Tiles Part 1 - Creation

Vector tiles contain vector representations of data across a range of scales. Unlike raster tiles, they can adapt to the resolution of their display device and even be restyled for multiple uses. With ArcGIS Pro 1.4 users can now also publish their data as vector tiles using local projections such as British National Grid. In this first of two posts, I'll take you through the steps required to publish your own vector tiles using a local projection.

Read More

The science behind our Coastal Views map

The science behind our Coastal Views map

When you're at the coast and gaze off into the horizon, do you know which countries are directly across the sea from you? I recently tried to answer this question by creating a map that went on to be featured on The Telegraph, Mail Online, BBC News and, more recently, Esri's Maps We Love gallery. 

Read More

Smart Mapping in ArcGIS Online

Back in March, Smart Mapping was added to ArcGIS Online. One of the aims is to make it easier to produce effective maps of your data, but it also adds sophisticated style options to ArcGIS Online. If you haven’t looked at it yet, or have only used a few bits, it’s worth taking a look through the series of blog posts on it.
Read More

10 minutes in the Marketplace

In September the ArcGIS Marketplace was launched. I thought I would spend 10 minutes to find out what I can do with it. In this time I was able to secure a free trial for a new reporting app and use open data through ArcGIS online to produce an interesting view on GCSE attainment in the UK.
Read More

Why are my OGC® Web Services displaced by about 120 metres?

You add your British National Grid data to a new Map Document in ArcMap and publish it to ArcGIS Server, ticking the Web Map Service (WMS) box, getting no errors and all is well; you even add the WMS to another Map Document  and everything matches perfectly.  Later when the user - you know the one who always finds things which go wrong - adds the WMS to their mash-up they call you and inform you that the map is shifted about 120 metres.  Sound familiar?

So you curse, kick the server (not literally, I hope) and get on the phone to Support to complain about a bug in the software.  Unfortunately the bug is not in the software, the omission is in the OGC® WMS Specification and potentially affects all WMS, WFS, WCS (in fact any W*S) Services. 

The user who reported the problem is not using British National Grid (BNG) as the Coordinate Reference System (CRS) of their map but something based on a different datum to GCS_OSGB_1936, which BNG is based upon, such as GCS_WGS_1984 or a projection based on a different datum such as WGS 1984 Web Mercator Auxiliary Sphere.  And the reason why the map is shifted is because no Geodetic Datum Transformation (GDT) has been applied to the published Map Document (see this blog entry for information about transformations).  But why would you expect to apply a GDT if you are only using BNG? 

We will skip the science bit about the W*S Specification until later but focus now on the workaround to overcome the problem that this omission in the OGC® W*S Specifications causes in the publishing of W*S using ArcGIS.

Workaround

After you have added your British National Grid data to the DataFrame in ArcMap you should change the DataFrame CRS to GCS_WGS_1984 and apply a transformation (Choose the transformation as per our earlier blog Understanding which Transformation to choose in ArcGIS)

After adding the transformation change the DataFrame back to British National Grid and save the Map Document, then publish to ArcGIS Server as normal.  If you want to add INSPIRE CRS capabilities to the WMS then a further 2 transformations would need to be added - as explained in the Science Bit.

Your problem is resolved!

The Science Bit – For those of you who wish to know.

The OGC® have been informed of the omission of Geodetic Datum Transformations in Web Services (Change Request 09-187 dated 22 December 2009 – Status Pending) but little has been discussed outside of OGC®.

If the native CRS of the data is based on any datum other than GCS_WGS_1984 then there should always be a GDT between that CRS and GCS_WGS_1984; this is required for accurate calculation of <EX_GeographicBoundingBox> attributes if nothing else. 

The formula to calculate the minimum number of GDT required for a WMS (assuming that each GDT is reversible) is:

where x is the different number of Datum required.

So let’s say you have British National Grid data and you need to show the WMS in Web Mercator and INSPIRE, then you have 3 datums and looking at the formula you require 3 GDT such as:

CRS Required in WMS

GDT

EPSG Code

British National Grid

GCS_WGS_1984

GCS_ETRS_1989

OSGB_1936_To_WGS_1984_7

OSGB_1936_To_ETRS_1989_1

ETRS_1989_To_WGS_1984

5339 *

5338 *

1149

 

*  Look at the blog entry OSTN02 supported in ArcGIS desktop to include these transformations in ArcGIS 10.0.

The following is a graph of the formula.

You have already been shown how to add a transformation between British National Grid and GCS_WGS_1984; to add the transformation between British National Grid and GCS_ETRS_1989 you need to have installed the OSTN02 in NTv2 and then change the DataFrame CRS to GCS_ETRS_1989 and apply the “only” transformation available.  To add the final GDT you need to add some GCS_WGS_1984 data while the DataFrame is in GCS_ETRS_1989 then add a transformation selecting GCS_WGS_1984 and the transformation to GCS_ETRS_1989.

Remove the added GCS_WGS_1984 data, change the DataFrame back to British National Grid, save the MXD and publish to ArcGIS Server as normal.

Jim Sibbald is looking at writing an Add-In for ArcGIS 10.0 to provide a simpler interface to the workaround.

Summary

When preparing W*S services with data other than that based on GCS_WGS_1984 you must always ensure that suitable Geodetic Datum Transformation(s) are added to the Map Document before publishing to ensure that you do not get the user problem displayed above.

Jim Sibbald

 

New ArcGIS Performance calibration tool

Do you wish to analyse the performance of your layers within a map? Are your layers taking longer to render than expected? Are you about to publish your maps to ArcGIS Server and you wish to optimise their performance?

If so, help is at hand. A new tool for ArcGIS 10 and ArcGIS 9.3.1 has recently been released which allow you capture performance information. The tool, PerfQAnalyzer, integrates into ArcGIS Map (or it can be run as a stand-alone ArcGIS Engine application) which helps by allowing you to analyse on a layer-to-layer basis the performance of your desktop application. Furthermore, the workflows captured within the tool are fully automatable and can be scripted and executed simply at the command prompt or within ArcMap.

The PerfQAnalyzer tool is a free, unsupported, downloadable tool. More information about the PerfQAnalyzer can be found in the following blog article http://blogs.esri.com/esri/supportcenter/2012/04/03/new-arcgis-performance-calibration-tool/ which also contains links to download the tool.

Working with Date Fields in Field Calculator

First thing first - I am not a developer.  Sounds a simple enough thing, but sometimes, when I want to work with date fields in my feature classes, it feels like I need to be.  If this statement resonates with you, then help is at hand.  If you spend hours staring at the field calculator blankly, trying to work out what Date ( ) and Now ( ) actually mean, then I share your pain.   

In my work, I frequently have to work with date and time data in tables; creating new data, reformatting existing data or performing analysis using this data.  The trouble is, I always forget the correct expression required to perform some of these calculations, and as the required syntax is not always straight-forward, I usually end up pestering my colleagues who have developer experience to help me out.

And so, as an aid-memoir to me and hopefully for the benefit of a few of you, I have listed below some of the most common challenges I face and examples of the expressions required to get the required results.  If this inspires you to try some more, then I have also listed some other resources (which I have used in preparing this list) at the bottom of this post.

Although my preference has been to write expressions in VB Script I’ve also included equivalent Python examples as well. Python is becoming more and more integrated within ArcGIS so if you’re not familiar with using it the examples I’ve included are an easy way to dip your toe in.

Example Data Calculations include:

  1. The difference between Shapefiles and Geodatabase date fields – not an expression, but very useful to understand before you carry any out!
  2. How to field-calculate today’s date
    • As Date and Time
    • As Date only
  3. How to field-calculate a specific date
  4. How to field-calculate random dates for a specific time period
  5. How to convert dates in a String/Text field into a Date field

1. The difference between Shapefiles and Geodatabase date fields

One of the first things to be aware of is a subtle, yet crucial, difference between the way a shapefile and a geodatabase store date values.

  • A shapefile (shp) stores dates in a date field with this format: yyyy-mm-dd.
  • A geodatabase (gdb, mdb or sde) formats the date as datetime yyyy-mm-dd hh:mm:ss AM or PM.

Therefore if your data contains times, then you must use geodatabases (file, personal or SDE) or your times will be truncated to dates.

Settings on your Windows system determine how the dates are displayed in ArcMap—M/d/yy, MM/dd/yy, yy/MM/dd, and so on. ArcMap uses the system short date format (numerical) for displaying dates. To alter your settings, go to Start>Control Panel>Region and Language

2. How to field-calculate today’s date

This may seem a simple task, but even this requires the correct Expression.  These expressions may be used to populate a Date field or a Text field.

a. As Date and Time (for Geodatabases only – expression will work for shapefiles but will return the date only)

Using VB Script

MyField = Now ( )

Using Python

MyField = datetime.datetime.now( )

b. As Date only

Using VB Script

MyField = Date ( )

Using Python

MyField = time.strftime("%d/%m/%Y ")

3. How to field-calculate a specific date

Sometimes, you have a specific date you want to populate numerous records with.  To do this, you simply surround the date with symbols as per below:

Using VB Script

Date format: #DD-MM-YYYY# or #DD-MM-YYYY HH:MM:SS#

Example expressions:

e.g. for dates only:                         

MyField = #30-01-2012#

 e.g. for dates and times:              

MyField = #30-01-2012 12:35:15#

Using Python

Date format: “DD-MM-YYYY” or “DD-MM-YYYY HH:MM:SS”

Example expressions:

e.g. for dates only:                         

MyField = "30-01-2012"

 e.g. for dates and times:              

MyField = "30-01-2012 12:35:15"

4. How to field-calculate random dates for a specific time period

This may not be an everyday requirement, but it is something I need to do a lot when creating fictitious demo data.  For example, the following code will create random dates between 01/01/2010 (inclusive) and 01/01/2011 (exclusive). In other words, the random dates go from 01/01/2010 to 31/12/2010, i.e. any date in 2010.

Using VB Script

Check “Show Codeblock”, and, in the “Pre-Logic Script Code” section, enter the following:

MinDate = #2010-01-01# MaxDate = #2011-01-01# Randomize Dim MinDate, MaxDate, RandDate RandDate = MinDate + INT((MaxDate - MinDate)*RND)

Edit the MinDate and MaxDate as appropriate. The MaxDate should always be the day after the maximum date you want to allow. So if you want to allow all dates in February 2010, “MinDate” should be 2010-02-01 and “MaxDate” should be 2010-03-01.

Then, beneath:

MyField = RandDate

date_image2.png

Using Python

Check “Show Codeblock”, and, in the “Pre-Logic Script Code” section, enter the following:

import random def randomDate(minDate, maxDate): minDate = datetime.datetime.strptime(minDate, "%Y-%m-%d") maxDate = datetime.datetime.strptime(maxDate, "%Y-%m-%d") diff = (maxDate - minDate).days return minDate + datetime.timedelta(random.randrange(0, diff))

Then, beneath:

randomDate("2010-01-01", "2011-01-01")

image3.png

5. How to convert dates in a String/Text field into a Date field

Often when data is provided to me or imported into ArcMap from Excel, dates are brought in as text fields rather than date field.  On some occasions, the dates are nicely formatted in one field whilst other times, the date may be spread over a number of fields.

If the date is already in a correctly formatted (e.g. ‘30/01/2012’ or ’30-01-2012’ or ’30 Jan 2012’) then the field may be directly calculated by referencing the field only as follows:

Using VB Script

Expression:                        

MyField = [StringDateField]

Using Python

Expression:                        

MyField = !StringDateField!

If, however the required date elements are split over numerous text fields or an element of the date is missing, the following expression style may be used which forms the date by adding each date element together:

Using VB Script

Example input 1:              3 fields formatted as 30 | 01 | 2012 (Day|Month|Year)

MyField = [Day]&"-"& [Month]&"-"& [Year]

Example input 2:              2 fields formatted as 30 | 01 (Day|Month) with no year

                                                let’s assume we want to set the year as 2009

MyField = [Day]&"-"& [Month]&"-"& 2009

Using Python

Example input 1:              3 fields formatted as 30 | 01 | 2012 (Day|Month|Year)

MyField = !Day! + "-" + !Month! + "-" + !Year!

Example input 2:              2 fields formatted as 30 | 01 (Day|Month) with no year

                                                let’s assume we want to set the year as 2009

MyField = !Day! + "-" + !Month! + "-" + "2009"

Resources

If that has left you wanting more, a good place to start are the following references I referred to:

The Field Calculator Unleashed:

http://www.esri.com/news/arcuser/0405/files/fieldcalc_1.pdf

Simplify Date and Time Calculations

http://www.esri.com/news/arcwatch/1210/tip.html

Fundamentals of Date Fields:

http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//005s00000018000000.htm

Date/Time Field manipulation in Python:

http://www.doughellmann.com/PyMOTW/datetime/

Useful Python Resources for ArcGIS:

http://blogs.esri.com/support/blogs/supportcenter/archive/2012/01/10/10-easy-ways-to-tame-python-scripting-in-arcgis.aspx

Tip: Set your region in ArcGIS.com for UK featured content

Earlier this month it was announced that there had been some updates to ArcGIS Online. I wanted to highlight one of these enhancements in particular which I believe will be of interest to UK based users.

It is now possible to localise ArcGIS Online for the UK. Defining your region sets the featured maps on the home page, content in the gallery, and the default extent of new maps in the ArcGIS.com map viewer to UK specific content.

The carousel on the ArcGIS.com home page will be populated with UK specific content as shown in the image below. Also, if you navigate to the Gallery then you’ll see that this also now contains UK Maps, Web and Mobile Apps.

Before seeing these changes you will need to modify your region in your ArcGIS.com profile. To do this follow these steps

  • Ensure that you’re logged into ArcGIS.com
  • Click on your name in the top banner to access your profile
  • In the Region options select “United Kingdom” (see below)
  • Click the Save button

Whilst you’re on your profile page, why not ensure that you’ve included some information about yourself and perhaps upload a photo of yourself?

The other major benefit to setting your region is that all new webmaps that you create with the ArcGIS.com map viewer or ArcGIS Explorer Online will initialise with the extent of the UK. This means that you wont have to zoom and pan to the region before starting to create your map.

The maps and other content that appear in the UK Featured Maps and the Gallery will continue to be refreshed so keep an eye out for new content. Also, if you’ve created a map that you wish to be included in these sections then please let us know. 

Shortcuts in ArcGIS Desktop

I'm currently in the midst of working on a series of four webinars entitled 'Getting the most out of ArcGIS'. These webinars are a response to customer feedback saying that they want to understand how to get more from ArcGIS desktop.  So we thought we would share some tips and tricks on areas we most commonly get questioned about. I would encourage you to watch it again here (and get my viewing figures up!) but if there was one top tip to remember it would be the following doc:

http://www.esri.com/library/brochures/pdfs/arcgis-desktop-tips.pdf

This is a document of shortcuts for saving you time in ArcMap.  An absolute must read for all users wanting to reduce their click count!

For those who have also been asking for access to the other links mentioned in the webinar, here they are... both excellent:

Desktop Blog:

http://blogs.esri.com/Dev/blogs/arcgisdesktop/archive/2010/08/10/Working-with-Microsoft-Excel-in-ArcGIS-Desktop.aspx

Geodatabase Blog:

http://blogs.esri.com/Dev/blogs/geodatabase/archive/2010/04/14/Query-Layers-_4000_-ArcGIS-10.aspx