Saturday, February 16, 2013

Offline Mapping in HTML5 Mobile Apps

One of our top feature requests for Plot Hound has been to add satellite/terrain layers to the cruise maps.  We couldn't just drop in normal Google or Bing Maps because we needed the maps to be available out in the woods with no data connection.

There are fairly straightforward ways to do this in native apps (like offline map caching in the Google Maps Android app and the MapBox iOS SDK), but there aren't great solutions for HTML5 apps built with PhoneGap.  And you have to stay away from using Google's map tiles because they have a pretty restrictive terms of service (MapBox seems to be the current leader in programmatic access to raw map tiles).  The best solution I've seen was by Scott Davis who used MapBox's TileMill to create an mbTiles SQLite database which he then accessed through the Cordova SQLite plugin and a custom Leaflet TileLayer.  The cool thing about this solution is that it leverages SQLite's built-in compression to store map tiles in a really efficient way.

But... there was a problem.  We're heavy users of PhoneGap Build to compile our PhoneGap app, and PhoneGap Build only supports a small set of plugins right now.  And no, the SQLite plugin isn't one of them.

Still, it seemed like I could take the basic structure of Scott's solution but just store the raw tiles locally with the PhoneGap Files API and then point a standard Leaflet TileLayer at the local directory.  My only concern was that the uncompressed raw tiles would be too big to reasonably download.  However, for our use-case (high-zoom on a relatively small area, low-zoom on the surrounding area), it seemed like we might be able to get away with it.  After a quick test in Python, I found that even doing zoom levels 3-17, we only were downloading about 600 tiles totaling about 6MB.  Good enough!

After hacking on it for two days, I came up with a working proof of concept.  The source is available on GitHub.  The instructions and technical details are in the README.md file.  I hope this code is a helpful start for developers trying to add offline mapping to their own apps - it was certainly a fun learning experience for me!

The final results:






34 comments:

  1. Awesome!

    Thanks for the shout out.

    ReplyDelete
  2. Congratulation, good work. I am also working on offline mapping on mobile (hopefully with phonegap based projects), have you take in consideration to build the images locally from row data using canvas or svg?

    ReplyDelete
  3. Hi Fabio - I haven't looked into doing that... I'm also not sure why you would want to, given that Leaflet does an incredible job already.

    ReplyDelete
    Replies
    1. Hi Max - I was thinking about it in order to solve the problem of heavy map tiles. If we can store the raw data instead of a lot of tile depending on zoom level, it would require a considerably less amount of space, don't you think? Of course, what I don't know is if it will require too much runtime computation..

      Delete
    2. Hi Fabio - what sort of "raw data" are you thinking of storing? Like storing the vector coordinates of the roads and then dynamically drawing that on a canvas? I mean, I guess it's possible, but I wouldn't want that job!

      Delete
  4. Awesome stuff.

    In this case the mbTiles database was created with TileMill but I guess this also works with other implementations for creating the .MBTiles file?

    Like the ones listed here
    https://github.com/mapbox/mbtiles-spec/wiki/Implementations

    Would be cool if I could use a WMS Server to download the maps instead of a MapBox ID.



    ReplyDelete
    Replies
    1. Hi Edward - so I didn't use an MBTiles file in this example (Scott did, as referenced in my links), I was using raw png images. You should be able to fork the code to pull tiles from any source.

      Delete
    2. Thanks man. I made some good progress on pulling tiles from my WMS but I have a question.

      Are you sure the pyramid of images is working as intended? For instance if you use [examples.map-vyofok3q] and zoom out a bit you can see Louisville and the edge of Lexington. But if you zoom in on Lexington it doesn't seem to have downloaded any images from those tiles in deeper zoom levels. So isn't the pyramid in reverse?

      Or am I missing something?

      Delete
  5. This comment has been removed by the author.

    ReplyDelete
  6. The way I wrote the tile-picking function... the center is hardcoded as Louisville and it downloads tiles within a radius of 1 for zoom 15 and below, 2 for 16, 4 for 17 (or something along those lines). If you have a different way you want to select tiles, you can modify the utils/tile.js -> pyramid function.

    ReplyDelete
  7. Thanks for this its what I need. To test I created a map terrycollinson.map-6vsly6sn and uploaded your zip file unedited to Phonegap Build. I installed on my device but when I tried to download using the new id I got an error - download error code 1. Not many moving parts so what can I have missed? Thanks

    ReplyDelete
    Replies
    1. That does appear to be a valid map id (see http://api.tiles.mapbox.com/v3/terrycollinson.map-6vsly6sn/1/0/0.png) - and I think error code 1 means that PhoneGap couldn't find the remote URL. Are you connected to the internet? The only other possibility I can think of is that you're getting a permission-denied when trying to write the downloaded file (but that shouldn't be happening if you're using my code). One way to debug is to turn on PhoneGap Debug and do console.log() for each URL that the app is trying to download. Then visit those URLs yourself to make sure that they're valid.

      Delete
    2. Still trying to get it to work. Even installed from here https://build.phonegap.com/apps/311874/share but the default map did not show and entering the example map gave download error 3. Using Samsung Galaxy II. Any ideas?

      Delete
    3. Hi Terry - there's no "example" map... your terrycollinson.map-6vsly6sn appears to be legit. It worked on my Galaxy Nexus. Are you entering "terrycollinson.map-6vsly6sn" into the "MapBox IDs" field?

      Delete
  8. I did the same with OpenLayers, so you can tile WMTS, TileCache, OSM, WMS and XYZ layers. It can actually cache all raster layers. It's pretty good and works really good.

    ReplyDelete
    Replies
    1. Nice! Do you use this: http://openlayers.org/dev/examples/offline-storage.html ?

      Delete
  9. Hi awesome work congratulations Max.. My concern
    I have a mbtiles locally and I need to load the tiles.. in the source you download the raw from a MapBox server .. I can use without downloading, use locally mbtiles?

    ReplyDelete
    Replies
    1. Hi Roberto - sounds like you should check this out: http://geospatialscott.blogspot.com/2012/04/phonegap-leaflet-tilemill-offline.html

      Delete
  10. Thanks for replaying...the problem with this solution is that it does not work with newer versions of Phonegap at least not to me .. sqlite plugin with BLOB fields ..
    I do not understand the concept that your solution is where the map loads

    ReplyDelete
    Replies
    1. Hi Roberto - not sure what you're confused about... basically the files are just downloaded to a directory on the local filesystem and then you point the tileLayer at that directory to show the tiles on the map

      Delete
  11. That files what type are.. mbtiles, sqlite3, png...in my project i have a bd.mbtiles but with sqlite_plugin can not load the tiles with cordova2.2.0 ...

    ReplyDelete
    Replies
    1. Yeah - I'm no sure about .mbtiles and sqlite. The whole point of my approach was to avoid dealing with sqlite.

      Delete
  12. You load the tiles from this way {map_id}/{z}/{x}/{y}.png...I just have a db.mbtiles .. how can i convert mbtiles in this files? It is possible?..Thanks for all the answer

    ReplyDelete
    Replies
    1. Hi Roberto, you'll want to use Scott's custom TileLayer, found here: https://github.com/stdavis/OfflineMbTiles/blob/master/www/js/TileLayer.MBTiles.js

      Delete
  13. Now we have many apps to use with GPS and maps with Android apps and others, Technology facilitates humans to investigate and learn

    Google Earth

    ReplyDelete
  14. Good job!!! Informative content. Keep up...Mobile App design

    ReplyDelete
  15. Got this working for a Cordova 2.7 project.

    Used plugin: https://github.com/pgsqlite/PG-SQLitePlugin-iOS

    Changed

    - DB file extension to .db

    - plugin api code in core.js, TileLayer.MBTiles.js (e.g. new window.sqlitePlugin.openDatabase(...), queries wrapped in transaction, result set is res.rows.item(), etc...)

    - Modified the plugin code SQLitePlugin.m to include BLOB handling code include base64 (using NSData+Base64.m)

    ReplyDelete
    Replies
    1. That's awesome Chris! I'm waiting for PhoneGap Build to support the SQLite Plugin before I dive into that. Do you have a proof of concept on GitHub or anything?

      Delete
  16. Thanks for sharing your work. I now need to add some interactivity to the offline map, so that a user can get info after clicking on a point. Problem with the tiles is that they are PNG, so no lat/ longs for me to get events triggered. Has anyone tried this? Any pointers?

    ReplyDelete
    Replies
    1. Hi Sautindogo - you're right, the PNGs are just the tiles. Leaflet is the wrapper around those tiles that exposes the geographic hooks. You can capture the lat/lng by listening to the map 'click' event. See the "dealing with events" section here: http://leafletjs.com/examples/quick-start.html

      Delete
  17. great blog post. Infos are very usefull and saves me huge amount of time which I spend on something else instead of searching posts like this. Thank you... educational technology in the classroom

    ReplyDelete
  18. Excellent info. Really useful stuff. Good job...Keep it up web design agency seattle

    ReplyDelete