Manually building a tile server (14.04)

這裡描述如何在Ubuntu Linux 14.04系統下，安裝、設定與調校所需的軟體，建立自己的tile server. (image tile server)

This page describes how to install, setup and configure all the necessary software to operate your own tile server. The step-by-step instructions are written for Ubuntu Linux 14.04 LTS (Trusty Tahr).

Software installation

The OSM tile server stack is a collection of programs and libraries that work together to create a tile server. As so often with OpenStreetMap, there are many ways to achieve this goal and nearly all of the components have alternatives that have various specific advantages and disadvantages. This tutorial describes the most standard version that is also used on the main OpenStreetMap.org tile server.

It consists of 5 main components: Mod_tile, renderd, mapnik, osm2pgsql and a postgresql/postgis database. Mod_tile is an apache module, that serves cached tiles and decides which tiles need re-rendering – either because they are not yet cached or because they are outdated. Renderd provides a priority queueing system for rendering requests to manage and smooth out the load from rendering requests. Mapnik is the software library that does the actual rendering and is used by renderd.

In order to build these components, a variety of dependencies need to be installed first:

sudo apt-get install libboost-all-dev subversion git-core tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev munin-node munin libprotobuf-c0-dev protobuf-c-compiler libfreetype6-dev libpng12-dev libtiff4-dev libicu-dev libgdal-dev libcairo-dev libcairomm-1.0-dev apache2 apache2-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-dev libgeotiff-epsg node-carto

Installing postgresql / postgis

On Ubuntu there are pre-packaged versions of both postgis and postgresql, so these can simply be installed via the Ubuntu package manager.

sudo apt-get install postgresql postgresql-contrib postgis postgresql-9.3-postgis-2.1

Now you need to create a postgis database. The defaults of various programs assume the database is called gis and we will use the same convention in this tutorial, although this is not necessary. Substitute your username for username in the two places below. This should be the username that will render maps with Mapnik.

sudo -u postgres -i
createuser username # answer yes for superuser (although this isn't strictly necessary)
createdb -E UTF8 -O username gis
exit

Create a Unix user for this user, too, choosing a password when prompted:

sudo useradd -m username
sudo passwd username

Set up PostGIS on the PostgreSQL database (again, substitute your username for username below):

sudo -u postgres psql
\c gis
CREATE EXTENSION postgis;
ALTER TABLE geometry_columns OWNER TO username;
ALTER TABLE spatial_ref_sys OWNER TO username;
\q
exit

Installing osm2pgsql

osm2pgsql is under active development and is best compiled from source.

mkdir ~/src
cd ~/src
git clone git://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql
./autogen.sh
./configure
make
sudo make install

Install Mapnik library

Next, we need to install the Mapnik library. Mapnik is used to render the OpenStreetMap data into the tiles used for an OpenLayers web map.

Build the Mapnik library from source:

cd ~/src
git clone git://github.com/mapnik/mapnik
cd mapnik
git branch 2.2 origin/2.2.x
git checkout 2.2

python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/truetype/
make
sudo make install
sudo ldconfig

Verify that Mapnik has been installed correctly:

python
>>> import mapnik
>>>

If python replies with the second chevron prompt >>> and without errors, then Mapnik library was found by Python. Congratulations! You can leave Python with this command:

>>> quit()

Install mod_tile and renderd

Compile the mod_tile source code:

cd ~/src
git clone git://github.com/openstreetmap/mod_tile.git
cd mod_tile
./autogen.sh
./configure
make
sudo make install
sudo make install-mod_tile
sudo ldconfig

Stylesheet configuration

Now that all of the necessary software is installed, you will need to download and configure a stylesheet.

For these instructions we will use OSM Bright, a good all-purpose stylesheet with a style reminiscent of popular webmaps. (If you’d like a map that looks exactly the same as the one on openstreetmap.org, you’ll need to use the openstreetmap-carto project which has its own installation instructions.)

Together with most modern stylesheets, OSM Bright is written in a stylesheet language called CartoCSS which is reminiscent of the CSS used for web design. Its main advantage is easy reading/writing, leading to rapid prototyping (using utilities such as Tilemill, Kosmtik or Sputnik). However, you do typically need to adapt third-party CartoCSS stylesheets for your own server setup, and then compile them to the XML required by Mapnik; that’s what we’ll do here.

We will use /usr/local/share/maps/style as a common directory for our stylesheet files and resources.

Download OSM Bright

To begin with, we need to download both the OSM Bright stylesheet, and also the additional data resources it uses (for coastlines and the like).  Again, substitute your username for username in the “chown” command below.

sudo mkdir -p /usr/local/share/maps/style
sudo chown username /usr/local/share/maps/style
cd /usr/local/share/maps/style
wget https://github.com/mapbox/osm-bright/archive/master.zip
wget http://data.openstreetmapdata.com/simplified-land-polygons-complete-3857.zip
wget http://data.openstreetmapdata.com/land-polygons-split-3857.zip
mkdir ne_10m_populated_places_simple
cd ne_10m_populated_places_simple
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_populated_places_simple.zip
unzip ne_10m_populated_places_simple.zip
rm ne_10m_populated_places_simple.zip
cd ..

We then move the downloaded data into the osm-bright-master project directory:

unzip '*.zip'
mkdir osm-bright-master/shp
mv land-polygons-split-3857 osm-bright-master/shp/
mv simplified-land-polygons-complete-3857 osm-bright-master/shp/
mv ne_10m_populated_places_simple osm-bright-master/shp/

To improve performance, we create index files for the larger shapefiles:

cd osm-bright-master/shp/land-polygons-split-3857
shapeindex land_polygons.shp
cd ../simplified-land-polygons-complete-3857/
shapeindex simplified_land_polygons.shp
cd ../..

Configuring OSM Bright

The OSM Bright stylesheet now needs to be adjusted to include the location of our data files. Edit the file osm-bright/osm-bright.osm2pgsql.mml in your favourite text editor, for example:

nano osm-bright/osm-bright.osm2pgsql.mml

Find the lines with URLs pointing to shapefiles (ending .zip) and replace each one with these appropriate pairs of lines:

"file": "/usr/local/share/maps/style/osm-bright-master/shp/land-polygons-split-3857/land_polygons.shp", 
"type": "shape"

"file": "/usr/local/share/maps/style/osm-bright-master/shp/simplified-land-polygons-complete-3857/simplified_land_polygons.shp", 
"type": "shape",

"file": "/usr/local/share/maps/style/osm-bright-master/shp/ne_10m_populated_places_simple/ne_10m_populated_places_simple.shp", 
"type": "shape"

Note that we are also adding “type”: “shape” to each one. (If you’re using nano, to save, now press ctrl-X and Y.)

Finally, in the section dealing with “ne_places”, replace the “srs” and “srs-name” lines with this one line:

"srs": "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"

Compiling the stylesheet

We now have a fully working CartoCSS stylesheet. Before Mapnik can use it, we need to compile it into XML using the command-line carto compiler. First of all, we use OSM Bright’s own preprocessor, which we need to edit for our setup:

cp configure.py.sample configure.py
nano configure.py

Change the config line pointing to ~/Documents/Mapbox/project to /usr/local/share/maps/style instead, and change dbname from osm to gis. Save and exit.

Run the pre-processor and then carto:

./make.py
cd ../OSMBright/
carto project.mml > OSMBright.xml

You now have a Mapnik XML stylesheet at /usr/local/share/maps/style/OSMBright/OSMBright.xml .

Setting up your webserver

Next we need to plug renderd and mod_tile into the Apache webserver, ready to receive tile requests.

Configure renderd

Change the the renderd settings by editing the /usr/local/etc/renderd.conf (you’ll need to do it as root via “sudo”) and change the following five lines, uncommenting (removing the ‘;’) when required. They are found in the [renderd], [mapnik] and [default] sections.

socketname=/var/run/renderd/renderd.sock
plugins_dir=/usr/local/lib/mapnik/input
font_dir=/usr/share/fonts/truetype/ttf-dejavu
XML=/usr/local/share/maps/style/OSMBright/OSMBright.xml
HOST=localhost

Create the files required for the mod_tile system to run (remember to change username to your user’s name):

sudo mkdir /var/run/renderd
sudo chown username /var/run/renderd
sudo mkdir /var/lib/mod_tile
sudo chown username /var/lib/mod_tile

Configure mod_tile

Next, we need to tell the Apache web server about our new mod_tile installation.

Using your favourite text editor, create the file /etc/apache2/conf-available/mod_tile.conf and add one line:

LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so

Apache’s default website configuration file needs to be modified to include mod_tile settings. Modify the file /etc/apache2/sites-available/000-default.conf to include the following lines immediately after the admin e-mail address line:

LoadTileConfigFile /usr/local/etc/renderd.conf
ModTileRenderdSocketName /var/run/renderd/renderd.sock
# Timeout before giving up for a tile to be rendered
ModTileRequestTimeout 0
# Timeout before giving up for a tile to be rendered that is otherwise missing
ModTileMissingRequestTimeout 30

Tell Apache that you have added the new module, and restart it:

sudo a2enconf mod_tile
sudo service apache2 reload

Tuning your system

A tile server can put a lot of load on hard- and software. The default settings may therefore not be appropriate and a significant improvement can potentially be achieved through tuning various parameters.

Tuning postgresql

The default configuration for PostgreSQL 9.3 needs to be tuned for the amount of data you are about to add to it. Edit the file /etc/postgresql/9.3/main/postgresql.conf and make the following changes:

shared_buffers = 128MB
checkpoint_segments = 20
maintenance_work_mem = 256MB
autovacuum = off

These changes require a kernel configuration change, which needs to be applied every time that the computer is rebooted. As root via “sudo”, edit /etc/sysctl.conf and add these lines near the top after the other “kernel” definitions:

# Increase kernel shared memory segments - needed for large databases
kernel.shmmax=268435456

Reboot your computer. Run this:

sudo sysctl kernel.shmmax

and verify that it displays as 268435456.

Loading data into your server

Get the latest OpenStreetMap data

Retrieve a piece of OpenStreetMap data in PBF format from http://planet.openstreetmap.org/.

If you need the entire planet file, you can do it by issuing the following command (again substitute your user for username below):

sudo mkdir /usr/local/share/maps/planet
sudo chown username /usr/local/share/maps/planet
cd /usr/local/share/maps/planet 
wget http://planet.openstreetmap.org/pbf/planet-latest.osm.pbf

Since the whole planet is at least 18GB when compressed, there are links to smaller country or state sized extracts on that page. However, many people will only need one country or city; you can download PBF files for these (‘extracts’) from download.geofabrik.de. We would recommend that you test with a much smaller area, and only move up to the full planet when you are confident your setup is working.

Importing data into the database

With the conversion tool compiled and the database prepared, the following command will insert the OpenStreetMap data you downloaded earlier into the database. This step is very disk I/O intensive; the full planet will take anywhere from 10 hours on a fast server with SSDs to several days depending on the speed of the computer performing the import. For smaller extracts the import time is much faster accordingly, and you may need to experiment with different -C values to fit within your machine’s available memory.

osm2pgsql --slim -d gis -C 16000 --number-processes 3 /usr/local/share/maps/planet/planet-latest.osm.pbf

Let’s have a look at your data import as it progresses. The first part of the osm2pgsql output looks scary, but is normal:

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Mid: pgsql, scale=100, cache=4096MB, maxblocks=524289*8192
Setting up table: planet_osm_nodes
NOTICE: table "planet_osm_nodes" does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes"
Setting up table: planet_osm_ways
NOTICE: table "planet_osm_ways" does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways"
Setting up table: planet_osm_rels
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"

Don’t be concerned by the NOTICE: entries above. All normal. Next, osm2pgsql will start reading the compressed planet file.

Reading in file: /home/user/planet/planet-latest.osm.bz2

As osm2pgsql reads the planet file it will give progress reports. The line below will refresh every few seconds and update the numbers in brackets. This part of the import takes a long time. Depending on your server, it will take between hours and days.

Processing: Node(10140k 200.1k/s) Way(0k 0.00k/s) Relation(0k 0.00/s)

As the import proceeds, first the Node number will increase untill they are all imported, then the Way number and finally the Relation number. The speed at which the import occurs depends strongly on the hardware available and if the node cache fits into ram. That said, if your node cache is big enough, you can probably expect the speed of nodes to be on the order of a few hundred k per second for nodes, a few 10s of k per second for ways and a few hundred per second for relations. If the node cache does not fit into ram or you have set the cache value to low, the process can be an order of magnitude slower. If you are importing the full planet, the import can take days. If you are only importing single countries, the time should be cut down to minutes or hours. Do not interrupt the import process unless you have decided to start over again from the beginning.

Processing: Node(593072k) Way(45376k) Relation(87k)
Exception caught processing way id=110802
Exception caught processing way id=110803
Processing: Node(593072k) Way(45376k) Relation(474k)

The exceptions shown above are due to minor errors in the planet file. The planet import is still proceeding normally.

The next stage of the osm2pgsql planet import process also will take between hours and days, depending on your hardware. It begins like this:

Node stats: total(593072533), max(696096737)
Way stats: total(45376969), max(55410575)
Relation stats: total(484528), max(555276)
Going over pending ways
processing way (752k)

The processing way number should update approximately each second.

Going over pending relations
node cache: stored: 515463899(86.91%), storage efficiency: 96.01%, hit rate: 85.97%
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_polygon
Committing transaction for planet_osm_point
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
Stopped table: planet_osm_nodes
Building index on table: planet_osm_ways
Stopped table: planet_osm_rels
Completed planet_osm_point
Completed planet_osm_roads
Completed planet_osm_polygon
Completed planet_osm_line
Stopped table: planet_osm_ways

Osm2pgsql took 86400s overall

The number of nodes, ways and relations processed will obviously differ by the size of the data extract you are using and the date of the data dump. The numbers shown here are not reflective of the full planet import, which is substantially larger.

Up and running

Testing your tileserver

Now that everything is installed, set-up and loaded, you can start up your tile server and hopefully everything is working. We’ll run it interactively first, just to make sure that everything’s working properly. Remember to substitute your username again:

sudo mkdir /var/run/renderd
sudo chown username /var/run/renderd
sudo -u username renderd -f -c /usr/local/etc/renderd.conf

and on a different session:

sudo service apache2 reload

If any FATAL errors occur you’ll need to double-check any edits that you made earlier.

If not, try and browse to http://yourserveraddress/osm_tiles/0/0/0.png to see if a small picture of the world appears. The actual map tiles are being created as “metatiles” beneath the folder /var/lib/mod_tile.

Setting it to run automatically

If it ran successfully, you can stop the interactive renderd process and configure it to run automatically at machine startup as a daemon.

sudo cp  ~/src/mod_tile/debian/renderd.init /etc/init.d/renderd
sudo chmod u+x /etc/init.d/renderd

Edit the /etc/init.d/renderd file as root – you’ll need to make a couple of changes to the DAEMON and DAEMON_ARGS lines so that they read:

DAEMON=/usr/local/bin/$NAME
DAEMON_ARGS="-c /usr/local/etc/renderd.conf"

Also, you’ll need to change references to www-data so that they match your username – change “www-data” to what you changed “username” to in other files.

You should now be able to start mapnik by doing the following:

sudo /etc/init.d/renderd start

and stop it:

sudo /etc/init.d/renderd stop

Logging information is now written to /var/log/syslog instead of to the terminal.

Next, add a link to the interactive startup directory so that it starts automatically:

sudo ln -s /etc/init.d/renderd /etc/rc2.d/S20renderd

and then restart your server, browse to http://yourserveraddress/osm_tiles/0/0/0.png and everything should be working! You can also go to the page http://yourserveraddress/mod_tile which should give you some stats about your tile server.

Congratulations. Head over to the using tiles section to create a map that uses your new tile server.

Acknowledgements

Originally based on material by Ian Dees and Richard Weait. Edited and updated by Richard Fairhurst with additional material by Andy Townsend and Paul Norman.

Source reference: https://switch2osm.org/serving-tiles/manually-building-a-tile-server-14-04/