The only extra program the script uses is GDAL. So I had to make sure GDAL/ogr2ogr was installed on my Ubuntu machine.
This script was actually my first Bash Script ever (not the batch downloader). It has a lot of bad practices that I have improved on since then, but in its current form works!
I will update the script once I get all my stuff settled into my place (moving...)
It will start by asking you for a workspace where it can create a folder called 'shp' and some subfolders: pt, ln and ply.
The zip file contents are then extracted to the shp folder.
Only the required shapefile parts are unzipped: .shp, .shx, .dbf, .prj
The script ignores the metadata or other unnecessary files for this process..
The shapefiles are then analyzed quickly for some feedback.
A single shapefile from each CanVec category is placed into a folder: pt, ln, ply
ogr2ogr requires a source shapefile to merge all other similar shapefiles to.
Source shapfiles are then appropriately renamed and the batch merging begins!
This script may bail if the shapefile size limit is reached (2-4gb) so I am considering other storage formats such as a postgresql database or a GRASS database for further geo-processing.
Note that merging in ogr2ogr does not create a seamless shapefile. Where two features come together, there will be a seam or segments. This is something I want to get rid of and will implement it probably through GRASS in the near future.
- Duplicate variables
- No string manipulation (all done with built-ins like cat rev cut etc...)
Be sure to set the permissions (make it an executable) prior to running it or it won't execute:
The script will provide lots of feedback during its course so scroll up in the terminal to get some interesting information when it completes.
Screen capture of CanVecSHP_v1.sh in Ubuntu terminal - Powered by Zoomify
960 zip files (3gb) unzipped in under 3 minutes.
Extracted 58,588 files (9gb)
Merged points(52mb), lines(7.6gb) and polygons(1.7gb) in about 13 minutes.
A 6gb shapefile took just under 4 minutes to merge and 20 seconds to merge 250mb.
Below are some screenshots of what the merged data looks like from way up.
Note that the CanVec data is not projected to BC Albers but in its native projection.