This page provides information about splitting up large OpenStreetMap files into a more suitable size. This is an optional data preparation step before using GO Loader to load OpenStreetMap data into your relational database.
Why use the OpenStreetMap Splitter Tool?
The OpenStreetMap Splitter tool enables Go Loader to operate in a more robust way, than it would loading a massive single file. By processing multiple smaller files one after the other, the data can be committed to the database regularly. Therefore should something happen unexpectedly, there would be no risk of losing all your data therefore needing to restart the load from scratch. Instead, you would only lose the data that was being processed at the time of a fail: a comparatively small data loss compared to one large file.
Go Loader generates a pass and fail list as it is loading data to enable you to simply 'pick up where you left off' and continue loading data, something worth taking advantage of should such a problem occur.
How 'large' is LARGE?
A 'large' amount of data i.e. more than a small city worth or > 1 million features will require the source .osm.bz2 file to be split into smaller, more manageable files (otherwise the transaction will be too large and your database will fail to commit the data). To put this into context, planet.osm.bz2 (the whole world!) is 18Gb compressed and has over 3 billion features in it, that's 3,000,000,000 features in 1 transaction if you don't split it! Obviously with a dataset this BIG, the OpenStreetMap Splitter tool would take an number of hours to split the 'Planet' datafile into smaller chunks - actual time taken is hardware dependant.
How to use the OpenStreetMap Splitter Tool
Use it to break very large osm.bz2 files into smaller chunks for easier processing and loading by GO Loader.
Step 1. Open the OpenStreetMap Tool:
From the GO Loader Tools menu select the OpenStreetMap Splitter Tool.
Image displaying how to open the OpenStreetMap Splitter Tool.
Step 2. Complete the OpenStreetMap Splitter dialog window:
Upon clicking the OpenStreetMap Splitter Tool button, an OpenStreetMap Splitter Tool dialog window opens.
This landing page explains what the OpenStreetMap Splitter Tool does.
Image displaying the OpenStreetMap Splitter GUI.
Step 3. Complete the OpenStreetMap Splitter Tool Options:
Locate the source OpenStreetMap file to be split
- Click on the Browse button to locate the file you wish to split.
- Select the file and Click OK.
Locate the output location for the split files.
- This can be auto populated by ticking the 'Generate Destination using Source OSM File' option.
- Alternatively click on the Browse button to locate the directory where you want split files to be placed.
- Click OK.
Number of files to skip
For use when you wish to resume splitting after an error has occured. Is dependant on how many files have been generated during the splitting process up until the error.
The default is 0
If an error does occur, for example your computer crashes at the 99th file, then you can put 99 into this field and all the files created up to the 99th file will be ignored. The 99th file will be created if it doesn't already exist, and if it has been partly created, this file will be overwritten.
Max features per file
The number of features that will be placed into each file.
The default value is 1,000,000 features
This produces a 15-25mb compressed .bz2 output file or an 150-250mb uncompressed files.
Number of parallel processes
Allows parallel processing of the splitting process and is hardware dependant. We recommend its use if you have available processors.
A checkbox to indicate whether output files will be compressed or uncompressed.
The default is checked so files will be compressed
Please note, that it will take time to compress files. Compressed files are much smaller and easier to handle.
Step 4. Start Splitting...
When all the parameters are completed the "Start Splitting..." button becomes enabled.
- Click the Start Splitting... button
- A new window is opened to display the progress of the OpenStreetMap file splitter.
Image displaying the Splitting Process progress bar.
If you click cancel the OpenStreetMap file splitter processing will stop and you will return to the previous screen.
- Click OK when the process is complete.
Now the process is complete...
- Click Done - this closes the OpenStreetMap Splitter tool window.
Your files have now been split successfully and are ready to be loaded. Find out how to load OpenStreetMap data into your database using GO Loader.