alternatives to RoyalMail©

• commercial data services
  • providers of address verification
    • cleansing, verification, demographic profiling, historical addresses
  • mailing providers
  • software providers (per user, per year) [1]
    • e.g. 5,000 transactions per month, 40£/month [2]
    • e.g. 5-10p per transaction [3]
  • address crawlers [4]
• public domain
  • FreeThePostcode.co.uk
  • New Popular Edition Maps - npemap.org.uk
  • free list of UK district codes (SW14) [5]
  • => together 1% postcode coverage

UK Postcodes

  SA3 1HD     LL29 7YG    AL10 8LE    LL29 8EY    
  LU6 2JQ     WF3 4HN     BH23 1JJ    DY12 1QQ
  [A-Z]{1,2}[0-9R][0-9A-Z]? [0-9][A-Z-[CIKMOV]]{2}

• one for every 15 delivery points (households)
  • "however this can vary between 1 and 100"
• 4 levels

      SW      -  area      -         124
      SA3     -  district  -       3,000
      SA3 1   -  sector    -      10,000
      SA3 1HD -  postcode  -   1,780,000
  

RoyalMail© products

• Postzon© data file
  • 500MB CSV file
  • 2.1 mio records, 1.8 mio postcodes
  • includes streetnames, boroughs, city names, counties
  • includes coordinates in UK Grid
    • "approximate centre of the area covered by the [post]code"
    • accurate to nearest cell (100m)
  • 50£ per user per year or 1,000£ per machine per year, 600£ minimum data supply cost of £600.
• other products [1]
  • data file with 1m accuracy
  • PAF: 27 mio full addresses and housenumbers
  • Alias file (street renames, historical data)

Geo::Postcode

• loads data file into SQLite

• $postcode = Geo::Postcode->new('SW1 1AA');
  if ( $postcode->valid ){
      print $postcode->lat, $postcode->long;
  }
  ($unit, $sector, $district, $area) = analyse('SW1 1AA');
  $distance_km = distance_between('LA23 3PA', 'EC1Y 8PQ');
  
  

• why donsn't nestoria use it?
  • no cleansing
  • core fragments/units functions offer little fallback
  • no notion of accuracy
  • not reusable for our query analysis

Geocoding

Let's geocode some data!

• Easy -- coordinate systems
• Medium -- postcodes
• Challenge -- streets
• Hard -- Text analysis
  
  
• Confidence levels

Geocoding - Easy

other coordinate systems

• convert UK Grid
  • Geo::Coordinates::OSGB (Ordnance Survey)
• check if result is inside the UK

Geocoding - Medium

UK postcodes

• cleanse, add spaces
• regexp match
• lookup in database
• fallback to sectors, districts

(((^[BEGLMNS][1-9]\d?) | (^W[2-9] ) | ( ^( A[BL] | B[ABDHLNRST] |
C[ABFHMORTVW] | D[ADEGHLNTY] | E[HNX] | F[KY] | G[LUY] | H[ADGPRSUX] |
I[GMPV] | JE | K[ATWY] | L[ADELNSU] | M[EKL] | N[EGNPRW] | O[LX] |
P[AEHLOR] | R[GHM] | S[AEGKL-PRSTWY] | T[ADFNQRSW] | UB | W[ADFNRSV] |
YO | ZE ) \d\d?) | (^W1[A-HJKSTUW0-9]) | (( (^WC[1-2]) | (^EC[1-4]) |
(^SW1) ) [ABEHMNPRVWXY] ) ) (\s*)? ([0-9][ABD-HJLNP-UW-Z]{2}))

Geocoding - Medium

Problems with geocoding postcodes

• Postal data
  • not error free
  • new houses, updates, deletions
  • sub-districts in London: W1F, EC2A
  • British Forces, Antarctica, Falkland, Saint Helena ...
• Incoming addresses
  • missing country field
    • we displayed Spanish vacation homes in Northern Ireland
  • crawler matching can fail if regexp too broad
    • non-postcodes get matched ("nz8")

Geocoding - Challenge

Street parsing

• about 800k streets, 350k unique names in UK
  • 1341x High Street
  • 1319x Station Road
  • 1015x Church Street
• we need district postcode to find list of streets
• pre-calculate center (mean) point of street
• abbreviations
  • Av, Dr, Rd, Sq, Pl, Pk ... (140)
• cleansing, typos, spacing, house numbers, etc

Geocoding - Hard

Full text analysis

• cleanse address and Royalmail© data
  • "Cornwall and Isles of Scilly"
  • "Glasgow, City Of"
  • "County of Herefordshire"
• identify postcode first
• identify counties
• then city/borough

Geocoding - Hard

random examples

Albert Villas, Gilbert Mews, LEIGHTON BUZZARD,
   Bedfordshire, LU7 1NF
Lake Lock Drive, Wakefield West Yorkshire
Church Lane, London
C ELMINGTON ROAD, LONDON
Fountain Road (Flat 2), Edgbaston
London SE18, UK
Lee, London, UK
Free Trade Wharf, 340 The Highway, Wapping/Limehouse London
Stunning 3 Storey Contemporary Barn Conversion In The Popular Village Of Utkinton
A.4-1 GREAT NORTHERN TOWER,WATSON STREET,CITY CENTRE,
   MANCHESTER GREATER MANCHESTER
15, ASTLEY HOUSE, LONDON

Measure confidence

• assign each method a confidence or accuracy
  • continuous improvement
• used for quality and NestoriaRank© [1]

Google geocoder confidence levels [2]

0 - Unknown location
1 - Country
2 - Region (state, province, prefecture, etc.)
3 - Sub-region (county, municipality, etc.)
4 - Town (city, village)
5 - Post code (zip code) <= in UK this is 10
6 - Street
7 - Intersection
8 - Address

Questions?

• mtm -at- lokku.com
• www.nestoria.co.uk/help/feedback