OSM – Loading data into your server

Get the latest OpenStreetMap data

Retrieve a piece of OpenStreetMap data in PBF format from http://planet.openstreetmap.org/.

If you need the entire planet file, you can do it by issuing the following command (again substitute your user for username below):

sudo mkdir /usr/local/share/maps/planet
sudo chown username /usr/local/share/maps/planet
cd /usr/local/share/maps/planet 
wget http://planet.openstreetmap.org/pbf/planet-latest.osm.pbf

Since the whole planet is at least 18GB when compressed, there are links to smaller country or state sized extracts on that page. However, many people will only need one country or city; you can download PBF files for these (‘extracts’) from download.geofabrik.de. We would recommend that you test with a much smaller area, and only move up to the full planet when you are confident your setup is working.

** Here you can download the data from http://extract.bbbike.org/ . By selecting the bound box and do select the format as pbf or any other according to your choice. You will get the email with the link having the data of the region you want and then download it .

Note : 1.Data is only Available for 48 hours to download from link you got in mail .

2. Do use the sudo in front of wget.

area

Importing data into the database

With the conversion tool compiled and the database prepared, the following command will insert the OpenStreetMap data you downloaded earlier into the database. This step is very disk I/O intensive; the full planet will take anywhere from 10 hours on a fast server with SSDs to several days depending on the speed of the computer performing the import. For smaller extracts the import time is much faster accordingly, and you may need to experiment with different -C values to fit within your machine’s available memory.

osm2pgsql --slim -d gis -C 1500 --number-processes 4 /usr/local/share/maps/planet/planet-latest.osm.pbf

Let’s have a look at your data import as it progresses. The first part of the osm2pgsql output looks scary, but is normal:

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Mid: pgsql, scale=100, cache=4096MB, maxblocks=524289*8192
Setting up table: planet_osm_nodes
NOTICE: table "planet_osm_nodes" does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes"
Setting up table: planet_osm_ways
NOTICE: table "planet_osm_ways" does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways"
Setting up table: planet_osm_rels
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"

Don’t be concerned by the NOTICE: entries above. All normal. Next, osm2pgsql will start reading the compressed planet file.

Reading in file: /home/user/planet/planet-latest.osm.bz2

As osm2pgsql reads the planet file it will give progress reports. The line below will refresh every few seconds and update the numbers in brackets. This part of the import takes a long time. Depending on your server, it will take between hours and days.

Processing: Node(10140k 200.1k/s) Way(0k 0.00k/s) Relation(0k 0.00/s)

As the import proceeds, first the Node number will increase untill they are all imported, then the Way number and finally the Relation number. The speed at which the import occurs depends strongly on the hardware available and if the node cache fits into ram. That said, if your node cache is big enough, you can probably expect the speed of nodes to be on the order of a few hundred k per second for nodes, a few 10s of k per second for ways and a few hundred per second for relations. If the node cache does not fit into ram or you have set the cache value to low, the process can be an order of magnitude slower. If you are importing the full planet, the import can take days. If you are only importing single countries, the time should be cut down to minutes or hours. Do not interrupt the import process unless you have decided to start over again from the beginning.

Processing: Node(593072k) Way(45376k) Relation(87k)
Exception caught processing way id=110802
Exception caught processing way id=110803
Processing: Node(593072k) Way(45376k) Relation(474k)

The exceptions shown above are due to minor errors in the planet file. The planet import is still proceeding normally.

The next stage of the osm2pgsql planet import process also will take between hours and days, depending on your hardware. It begins like this:

Node stats: total(593072533), max(696096737)
Way stats: total(45376969), max(55410575)
Relation stats: total(484528), max(555276)
Going over pending ways
processing way (752k)

The processing way number should update approximately each second.

Going over pending relations
node cache: stored: 515463899(86.91%), storage efficiency: 96.01%, hit rate: 85.97%
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_polygon
Committing transaction for planet_osm_point
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
Stopped table: planet_osm_nodes
Building index on table: planet_osm_ways
Stopped table: planet_osm_rels
Completed planet_osm_point
Completed planet_osm_roads
Completed planet_osm_polygon
Completed planet_osm_line
Stopped table: planet_osm_ways

Osm2pgsql took 86400s overall

The number of nodes, ways and relations processed will obviously differ by the size of the data extract you are using and the date of the data dump. The numbers shown here are not reflective of the full planet import, which is substantially larger.

Advertisements

Published by

kaurdavinder

I am the one who discover myself daily through reading, writing, interacting with people and expressing my thoughts to inspire them if I would be able to bring a little change to the society through my work. I write blogs, participate in various social activities and want to be a Professional Speaker. Currently, I am pursuing my post graduation in Computer Applications from Guru Nanak Dev Engineering College, Ludhiana and I’m very much active in many technical & social communities.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s