<h4>Dynamic CloudFront origin selection based on custom HTTP headers and cookies.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zUytg-OYhrNH9ghY" /><figcaption>Photo by <a href="https://unsplash.com/@gcalebjones?utm_source=medium&amp;utm_medium=referral">Caleb Jones</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure>Using S3 and CloudFront for storing and serving your single-page application is a no-brainer. For some cases this is sufficient. Once your application gains traction and you need to fix bugs or deliver updates and features, creating a more flexible environment is a sensible step to take. It makes sense to approach this flexible setup early on before launching your product. However, implementing it further down the road is not a problem at all.So what are you learning in this article?<ul><li>How to create pull-request previews which can be accessed by using a custom HTTP header</li><li>How to easily switch between deployments based on the git hash (which can be leveraged as a rollback mechanism) without any downtime</li><li>Get a more profound knowledge of the CloudFront caching mechanism</li><li>Get to know lambda@edge and the different events, lambdas can be triggered by</li></ul>This might sound complicated and complex but it actually is quite straightforward to implement and maintain. Let’s get right to it. I’ll use the AWS Console instead of Terraform to make this concept available for everyone. If you’re familiar with Terraform, the steps down below should be easily translatable into code.<h4>The Base setup – CloudFront with a fixed S3 origin</h4>Let’s start with the basic setup. As mentioned before, to just serve static HTML, we use CloudFront with an S3 bucket as its origin. (Sure, to make this nice for the user we also need IAM roles, Route53, etc. For the sake of simplicity and to keep this article as concise as possible, let’s only focus on the relevant parts).<figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*epdnctPdRWBOceuN8_MUjQ.jpeg" /><figcaption>Most basic AWS infrastructure setup for serving application</figcaption></figure><h4>Pull-Request Previews — Dynamically switching the CloudFront origin via lambda@edge and HTTP-Headers</h4>Let’s begin with setting up the infrastructure for the pull-request preview mechanism. The idea is to deploy each pull-request build into a separate bucket (e.g. via Codebuild and Codepipeline or any other CI/CD) and target a specific bucket by its pull-request number. To achieve that we need a lambda@edge function that evaluates the request headers and sets the new origin for the CloudFront distribution to the supplied pull-request number. As a custom header, I’ve decided to use x-myapp-origin, pr- as a prefix for the value and the number of the pr as an identifier. This ultimately translates to e.g. pr-1012.Lambda@edge functions can be associated with different CloudFront events: Viewer Request, Origin Request, Origin Response, and Viewer Response. In our case, it makes sense to associate our lambda with the Origin Request event as we want to switch over our origin. Further down I’ll cover the different events briefly. If you want to know more about the different events, head over to the <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-cloudfront-trigger-events.html">AWS docs</a> which illustrate the difference and some specific use-cases.<figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*lmuAr1JQHoStvGHBInw9Vg.jpeg" /><figcaption>Updated AWS infrastructure setup for a temporary origin switch</figcaption></figure>In case you don’t want to create a new bucket for each deployment, you can also achieve the same result by just changing the path of the respective target directory. You could use the same naming convention as with separate buckets. Keep in mind that you need to clean up the data yourself once your pull-request has been merged. Luckily there’s a webhook provided by Github to tell you when a pull-request has been merged.Below you’ll find the lambda@edge code to set up the pull-request preview. Just put this in a lambda with nodeJs as a runtime and associate it with the Origin Request event of your CloudFront distribution. To test this, <a href="https://requestly.io">requestly.io</a> is a handy browser extension to send custom headers.Hint: As I want to keep the focus of this article on the origin switch, in case you’re not comfortable with lambda@edge, you can get a brief overview on how to attach it to the Cloudfront distribution in the <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-edge-how-it-works-tutorial.html">AWS docs</a>.<iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd27a369155602e38085a3fb9e1996ea/href">https://medium.com/media/fd27a369155602e38085a3fb9e1996ea/href</a></iframe><h4>Rollbacks and Blue/Green Deployments — Dynamically switching the CloudFront origin via lambda@edge via cookies</h4>Now that we’ve seen how we can temporarily set a new origin for a CloudFront distribution by using HTTP headers, we’ll implement a more permanent solution using cookies (obviously we can’t expect our uses to send a custom header each time a new deployment is available).To differentiate between the different deployments, this time we don’t use the pull-request number but rather rely on the unique short git hash. On each deployment, we create a new bucket with the git hash being part of the bucket name e.g. web-bucket-a9624c57 . After each deployment, we save the git hash to the Parameter Store of the AWS Systems Manager to keep track of the latest deployment. To handle the cookie accordingly, we additionally need a Viewer Request and Origin Response lambda function association. Each of those function associations has its own purpose. The following excerpt is taken from the <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-cloudfront-trigger-events.html">AWS docs</a> and illustrates the different use-cases and capabilities:<blockquote>Viewer Request</blockquote><blockquote>The function executes when CloudFront receives a request from a viewer, before it checks to see whether the requested object is in the CloudFront cache.</blockquote><blockquote>Origin Request</blockquote><blockquote>The function executes only when CloudFront forwards a request to your origin. When the requested object is in the CloudFront cache, the function doesn’t execute.</blockquote><blockquote>Origin Response</blockquote><blockquote>The function executes after CloudFront receives a response from the origin and before it caches the object in the response. Note that the function executes even if an error is returned from the origin.</blockquote><blockquote>Viewer Response</blockquote><blockquote>The function executes before returning the requested file to the viewer. Note that the function executes regardless of whether the file is already in the CloudFront cache.</blockquote>I’ll explain the new updated infrastructure momentarily, for now, here it is.<figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*vhQulr1SUlnybEeLBQ23oQ.jpeg" /><figcaption>Updated AWS infrastructure setup for a permanent origin switch</figcaption></figure>Let’s go through the process step by step, starting with the user requesting the page. You’ll find some additional code in the lambda functions which is for handling the custom HTTP header for the pull-request preview.<ol><li>Viewer Request</li></ol>The users&#39; request is first evaluated in the lambda@edge function which is tied to the Viewer Request trigger. This lambda is invoked on each request as the user requests a file and CloudFront needs to check whether there’s already a cached version of the file available that can be returned to the user. Within that function, we<ul><li>Read the latest git hash from the parameter store</li><li>Check if the request contains a cookie and whether that cookie value is the same as the previously fetched git hash.</li><li>If that’s the case, we don’t alter the request and just pass it through to CloudFront.</li><li>If the hash is different, we update the cookie value to reflect the latest git hash and pass the request further down to CloudFront.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52e83287b92c3cae538f29dd9585e82b/href">https://medium.com/media/52e83287b92c3cae538f29dd9585e82b/href</a></iframe>2. CloudFrontCloudFront checks, based on the criteria for its caching mechanism, whether the requested file is already present at the edge location or whether it should be fetched from the origin. In case a previous user has already requested the file with the same cookie value in the same region, chances are high the next user gets the cached file.3. Origin RequestCloudFront couldn’t find a cached version and therefore needs to fetch it from the origin. We also registered a lambda function to be triggered in this case. The lambda can be broken down into the following steps:<ul><li>Read the latest git hash from the parameter store</li><li>Update the new origin based on the git hash</li><li>Return the request</li></ul>As you can see, the above steps are pretty straightforward. We don’t even take the cookie which has been set in the Viewer Request into consideration as it only acts as a caching key.<iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48a87696c23a15d5570ac2cf6f4c6454/href">https://medium.com/media/48a87696c23a15d5570ac2cf6f4c6454/href</a></iframe>4. Origin ResponseIn the Origin Response lambda, we‘re just dealing with the cookie. We need to explicitly set the cookie here so that the browser will set the cookie for the user. This way the cookie will be present in the next Viewer Request and therefore CloudFront will return a cached version. Here are the steps in detail:<ul><li>Read the latest git hash from the parameter store</li><li>Set the cookie</li><li>Pass the response to CloudFront</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d072b15cf3f81d9d448d37a4145249f4/href">https://medium.com/media/d072b15cf3f81d9d448d37a4145249f4/href</a></iframe><h4>Update the CloudFront Distribution Caching behavior</h4>As already mentioned before, CloudFront needs to consider the cookie when deciding which requests to cache. By default, CloudFront only uses the domain e.g. d111111abcdef8.cloudfront.net and the path of the requested object e.g. /content/stories/example-story.html as a cache key. To customize the behavior, head over to your distribution, click on the Behavior tab, select the behavior you want to edit (most probably there’s only one), and click on edit. After scrolling down, you’ll find the Cache key and origin requests section.<figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*Zge1HQwsF39_ko6R4yy9MA.png" /><figcaption>Cache key customization of the CloudFront distribution.</figcaption></figure>The screenshot above is the legacy view of the cache key configuration. Nowadays it makes sense to create a separate policy and attach it to the behavior. Strictly speaking, we need two policies. Let me elaborate. There are two types of policies in CloudFront. Cache Policies and Origin Request Policies. Even though they are grouped in the UI they are doing different things. The Cache Policy defines the criteria for the cache key, meaning what’s being taken into consideration to cache a request and therefore prevent a roundtrip to the origin. The Origin Request Policy is not related to caching even though it has the same options as the Cache Policy. However, this Origin Request Policy just defines which data is being passed on to the origin. So in case you need to read a cookie value or HTTP header in either the Origin Request lambda or the Origin Response Lambda, you need to create a custom Origin Request Policy.To create a policy, toggle the menu bar on the left inside CloudFront and select Policies, and click Create cache policy, enter a name and a description, scroll down to Cache key settings. Choose the headers as shown in the legacy settings screenshot from above. The same goes for cookies. The other defaults are just fine. So click Create to finish the Cache Policy creation. Next, click the Origin Request Policy tab and do the same as in the Cache Policy. Once you&#39;re done, head back over to the Behavior tab inside your CloudFront distribution, select the behavior you want to update, and scroll down to the Cache key. You can choose the policies you&#39;ve just created as the Cache Policy and the Origin Request Policy. Now you need to save your changes and wait for the updates to be applied.To be sure everything works as expected, just head over to your page and check two things:<ol><li>Has my previously defined cookie been set</li><li>Evaluate the custom HTTP response header x-cache sent by CloudFront. The first time it’s gonna be a Miss from CloudFront. All subsequent requests should be a HIT from CloudFront . If that&#39;s not the case, head back over to your caching policies and double-check them.</li></ol><h4>How to rollback</h4>Even though the idea behind this should be clear, just for the sake of completeness: Doing a rollback is done by setting a previous git hash in the Parameter Store. Make sure you keep the S3 buckets around as long as you might need them. I keep my last three deployments around, just in case.The whole process described above can be easily integrated into any kind of CI/CD. I’d recommend using AWS Codebuild and Codepipeline.<h4>Advantages and drawbacks of the described solution</h4>As always in programming, each solution has its pros and cons. Taking other mechanisms into consideration, I’d say the above solution is well worth having a look at and tinkering with it.Positive aspects<ul><li>Easy to set up</li><li>Barely any costs upfront</li><li>Easy to maintain (once you’ve set up your pipeline accordingly)</li><li>No downtime while switching between deployments</li><li>Even though a lambda@edge function is more expensive than a regular lambda function, it is still quite affordable (at least for the majority of use-cases)</li><li>You’ll learn a lot about CloudFront and S3</li></ul>Negative aspects<ul><li>Deviating from CloudFronts default caching behavior can have negative consequences in comparison to the default cache key in terms of performance</li><li>The same goes for lambda@edge which kinda slows down the request in case it’s not yet cached</li><li>The rollback mechanism is not coupled to the backend</li></ul><h4>Why are you not using CloudFront Functions?</h4>CloudFront functions are the new kid on the block and seem to be better suited for the above scenario. Unfortunately, CloudFront functions can only be triggered by the Viewer Request and Viewer Response.<h4>Where to go from here</h4>You’ve seen that with the approach above you can do all sorts of things. If you’ve integrated end-to-end tests into your workflow, this would be a perfect use case. You can run the e2e-tests first and once everything is green, you update the origin of the production environment to the successfully tested git hash.The same mechanism could also be used for A/B testing. If you can come up with other use-cases I’d be thrilled to hear about them in the comments 💬So long, happy coding 👌<h4>Resources</h4><ul><li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-cloudfront-trigger-events.html">CloudFront Events That Can Trigger a Lambda Function</a></li><li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions.html">Customizing at the edge with functions</a></li><li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/controlling-the-cache-key.html">Controlling the cache key</a></li><li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/controlling-origin-requests.html">Controlling origin requests</a></li><li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-edge-how-it-works-tutorial.html">Tutorial: Creating a Simple Lambda@Edge Function</a></li></ul>More content at <a href="http://plainenglish.io/">plainenglish.io</a><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7658ad0a7d5" width="1" height="1" alt=""><hr><a href="https://aws.plainenglish.io/rollbacks-and-pull-request-previews-using-cloudfront-lambda-edge-without-any-downtime-e7658ad0a7d5">Rollbacks and Pull-Request Previews using CloudFront &amp; lambda@edge without Any Downtime</a> was originally published in <a href="https://aws.plainenglish.io">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Rollbacks and Pull-Request Previews using CloudFront & lambda@edge without Any Downtime

<h4>Implement smarter routing using the Levenshtein Distance Calculation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BEqr5ZDpR6vEh50F" /><figcaption>Photo by <a href="https://unsplash.com/@visuals?utm_source=medium&amp;utm_medium=referral">visuals</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure>There are countless ways to improve the User Experience for your customer. Especially dead ends are a huge bummer for everyone. It’s not only about frustration but also the decrease of your conversion rate. Let’s try to prevent that and make your application or website event better by trying to redirect the user to the correct page instead of a 404 page.At the end of this article, you’ve learned how the Levenshtein Distance Calculation works, how it helps to correct typos, and how you can use it to prevent the user from being stuck on a 404 page without further guidance. First, we’ll implement the solution in plain JavaScript and later on as an Angular Route Guard.Side note: If you now think typos are not a big deal, every Macbook Pro user will be thanking you forever. Thanks to the great butterfly keyboard. 🦋<h3>What’s the Levenshtein Distance Calculation?</h3>Let’s first get a high-level understanding of what’s the Levenshtein Distance Calculation actually is. To keep things a bit more readable throughout the article I’ll substitute “LDC” for “Levenshtein Distance Calculation”.The LDC is a way to determine the similarity of two strings. It calculates a number that represents the required number of modifications to turn one string into another. The higher the number, the more edits are necessary and therefore there’s less similarity between the two input strings.If you take for example the two input strings kitten and sitting, the minimum number of required changes is three:kitten → sitten (substitution of “s” for “k”)sitten → sittin (substitution of “i” for “e”)sittin → sitting (insertion of “g” at the end).The LDC is a great method for correcting user inputs in command-line tools like git.Most likely the user only got a few letters mixed up and Levenshtein is there to help. The git CLI suggests the command you probably aimed for based on string similarity.So without compute-intensive machine learning, it’s possible to anticipate the users desired input. It’s quick, cost-effective, and simple to implement. As this is a simple string comparison without any artificial intelligence involved, this only helps in the case of typos. In case you’ve renamed one URL to another, for example, Preferences to Settings, LDC is not the right solution.If you want to know more about the calculation itself or the theory behind it, <a href="https://medium.com/@ethannam">Ethan Nam</a> has got you covered with his great article <a href="https://medium.com/@ethannam/understanding-the-levenshtein-distance-equation-for-beginners-c4285a5604f0">Understanding the Levenshtein Distance Equation for Beginners</a>. Besides the Levenshtein, there are a handful of other string similarity algorithms e.g. <a href="https://en.wikipedia.org/wiki/Cosine_similarity">Cosine</a>, <a href="https://en.wikipedia.org/wiki/Trigram">Trigram</a>, <a href="https://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance">Jaro–Winkler</a>, and <a href="https://en.wikipedia.org/wiki/Hamming_distance">Hamming</a>. For each use case, it makes sense to evaluate which algorithm is a fit as the approach of how the similarity is calculated differs. Sometimes the order is taken into consideration (e.g. Levenshtein) and sometimes it’s not (e.g. Cosine)For now, let’s go one step further and use it in an application to suggest the next best route in case a customer ended up on a 404 page because of a spelling mistake in a deep link e.g. misspelling privacy.<h3>Using the Levenshtein Distance Calculation to make route suggestions</h3>Now that we know what the LDC does, let’s implement a basic route suggestion mechanism to help the user decide where to go next in case the customer hits a 404, provoked by a typo. The idea is to display the most likely route the customer could have wanted to navigate to or redirect instantly.JavaScript is my choice of weapon and npm offers great implementations of the LDC. I’ve chosen to use the <a href="https://www.npmjs.com/package/fastest-levenshtein">fastest-levenshtein</a> library in this case.<a href="https://www.npmjs.com/package/fastest-levenshtein">fastest-levenshtein</a> exposes two functions, distance(a: string, b: string): number which takes two arguments, both are strings that are being compared and the result which is the calculated distance as a number. The second one is closest(str: string, arr: string[]): string which takes a string and an array to compare the string to. The result of the function is the string that requires the lowest number of modifications. For the sake of simplicity, we&#39;ll use the closest function. As probably no one is using plain JavaScript these days, I&#39;ll just stick to the logic (which is obviously quite slim as the <a href="https://www.npmjs.com/package/fastest-levenshtein">fastest-levenshtein</a> library is doing all the heavy lifting for us).<iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b0712e3179935b38696c9d1c71b1c1e/href">https://medium.com/media/9b0712e3179935b38696c9d1c71b1c1e/href</a></iframe>The first approach has its downsides. In the beginning, we just check if the pathname of the current route is present in our registered routes object. If not, we invoke the LDC to get the closest possible route. So far so good. But you might already have noticed, that this might cause some trouble. Let me elaborate on that. Imagine the following scenario: the user types in privacyy but you have renamed the privacy route to privacy-settings. LDC outputs a value of 9. But there’s another route present which is called profile. LDC outputs a value of 6. Now we’re screwed. As you can see, in this case, the algorithm doesn’t help at all. It might even confuse the customer as to how he/she’s been redirected to a page with a completely different context.Let’s try to optimize the first approach with some tweaks. This time we’ll use the distance function of the library to get more details about how close the current path is in comparison to all our registered routes. We also introduce the constant MAX_DISTANCE. As already seen in the first approach, we check if the current pathname is present in our registered routes object. If that’s not the case we use the LDC distance method to get the number of required modifications of the current pathname and each registered route. Then we sort the results by their number of modifications and filter them by the newly introduced constant MAX_DISTANCE. This way we can at least make sure that we only accept a reasonable route suggestion with only a few required modifications. Sure, this is also not bulletproof either but much better and sufficient for most cases.<iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af15334e25ad693ec3c897b3b0144631/href">https://medium.com/media/af15334e25ad693ec3c897b3b0144631/href</a></iframe><h3>Using the Levenshtein Distance Calculation to automatically redirect the user to the correct page with Route Guards in Angular</h3>To take the whole concept one step further, we implement a route guard in Angular, which automatically redirects the user to the route suggested by the LDC and therefore prevents the customer from seeing a 404-page altogether. Let’s get right to it. The full repo can be found <a href="https://github.com/danielreiser/medium-levenshtein-404-redirect">here</a>.<iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa41f737609812abcbd43beb14b737cd/href">https://medium.com/media/aa41f737609812abcbd43beb14b737cd/href</a></iframe>You’ve seen, with just a few lines of code (ok, to be fair, the <a href="https://www.npmjs.com/package/fastest-levenshtein">fastest-levenshtein</a> library does all the heavy lifting for us) and minimal time effort, you can minimize your user&#39;s frustration and improve the UX of your application.<h4>Resources</h4><ul><li><a href="https://medium.com/@ethannam/understanding-the-levenshtein-distance-equation-for-beginners-c4285a5604f0">Understanding the Levenshtein Distance Equation for Beginners</a> by <a href="https://medium.com/@ethannam">Ethan Nam</a></li><li><a href="https://en.wikipedia.org/wiki/Levenshtein_distance">Wikipedia Article about Levenshtein distance</a></li><li><a href="https://www.npmjs.com/package/fastest-levenshtein">fastest-levenshtein npm package</a></li><li><a href="https://github.com/danielreiser/medium-levenshtein-404-redirect">Angular Router Guard example</a></li><li><a href="https://en.wikipedia.org/wiki/Cosine_similarity">Cosine Similarity</a></li><li><a href="https://en.wikipedia.org/wiki/Trigram">Trigram</a></li><li><a href="https://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance">Jaro–Winkler Distance</a></li><li><a href="https://en.wikipedia.org/wiki/Hamming_distance">Hamming Distance</a></li></ul>More content at <a href="http://plainenglish.io/">plainenglish.io</a><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5c38359a294" width="1" height="1" alt=""><hr><a href="https://javascript.plainenglish.io/improve-your-ux-in-under-10-minutes-f5c38359a294">Improve Your UX in Under 10 Minutes</a> was originally published in <a href="https://javascript.plainenglish.io">JavaScript in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Improve Your UX in Under 10 Minutes

<h4>Create and host your own stylable vector tiles to make maps that stand out</h4><figure><img alt="Pins in a map" src="https://cdn-images-1.medium.com/max/1024/0*9xSVpBnAS0ZqZ07A" /><figcaption>Photo by <a href="https://unsplash.com/@teckhonc?utm_source=medium&amp;utm_medium=referral">T.H. Chia</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure>In this article, I’ll introduce you to the toolset and process to create vector tiles and show you how you can host the map data statically and cheaply without the need for a server. In the end, you’ll be able to use the map tiles in web-based products and mobile apps.Maps are cheap and most of the time — free. Not quite. There are situations in which you can use Google Maps or Mapbox complete for free or at least at a reasonable price.However, there are some contexts in which maps become super expensive and therefore are not an option for small businesses.Furthermore, especially if your product or site is targeted at EU customers, GDPR is always an obstacle you have to overcome. Hosting the tiles yourself makes all of the above less of an issue.Maps have become an essential part of our daily lives. Chances are high that you’ve either encountered maps, which are being powered by Google Maps, Apple Maps, or Mapbox.Nowadays, most maps (if not all) are some kind of derivative of data collected by the OpenStreetMap community.When I started working on <a href="https://aerwork.app">Aerwork</a>, I already knew that I needed maps to visualise flight paths and display airport locations.TL;DR If you don’t care about how I got to my final result, you can skip ahead and go directly to “<a href="#2312">The Way To Go</a>”: a step-by-step guide on how to convert, serve, and display your own map.<h3>Approach #1. Use Any Commercial Mapping Platform Like Mapbox or Google Maps</h3>As a developer with a strong visual focus, it was a no-brainer to head over to Mapbox, install their JavaScript SDK, design some beautiful maps in Mapbox Studio, and I&#39;m done (more or less, no flight paths yet but the basic setup is done).When I looked at their licensing page, my initial euphoria quickly vanished. For each web seat (if I provide maps to authenticated users), I would have to pay $4. For a SaaS without a single paying customer and a Plus tier starting at only $8, it seemed to be a dealbreaker. All other mapping providers offer a product in a similar price range and therefore did not make it into the product either.<h3>Approach #2. Host the Map Tiles Ourselves</h3>Next, I thought about providing the map tiles by myself. I mean it can’t be that hard. Raw OSM data is available for free under an ODbL license and therefore is allowed to be used in a commercial product.Well, things got ugly quite fast. Raw OSM data is insanely detailed and therefore huge. Trashcans, trees, buildings, airports, you name it. It’s there. Kudos to all OSM contributors.At the same time, this is an issue if you want to work with the data. It turned out that it is a rather ambitious plan to do everything yourself as there’s no “Just add water” solution. After doing some research I figured out that I needed an MBTiles file that contains the map tiles for the specific area of the world (or the whole planet) I am interested in.An MBTiles file is a format for storing map tiles in a single file. Technically it’s just an SQLite database. By using an MBTiles file you’re able to style the map using the <a href="https://docs.mapbox.com/mapbox-gl-js/style-spec/">Mapbox GL style specification</a>, which is a huge benefit as there are some editors out there to visually define the map styles e.g., <a href="https://maputnik.github.io/">maputnik</a>.So I needed to get my hands on some MBTiles. <a href="https://data.maptiler.com/downloads/planet/">Maptiler</a> (which is another mapping platform) offers already prepared MBTiles for certain areas or even the whole planet.You can subscribe and get regular updates or pay a one-time fee to download the MBTiles file for the required area. As stated above, Aerwork is a SaaS just starting out. I didn’t want to spend hundreds or even thousands of dollars along the way just to provide nice-looking maps. I’d love to do that but that’s far away in the future.Even though Europe would be sufficient for now, I couldn’t find any MBTiles file of Europe for free (or at least a bit cheaper than what maptiler is offering).So the plan stands. I create the tiles myself. After stumbling across the open source toolset that maptiler provides for this exact task of converting raw osm data to a usable MBTiles file, I’ve pulled the docker image from the <a href="https://github.com/openmaptiles/openmaptiles">openmaptiles</a> repo, followed along with their documentation and started by testing it with Hamburg.Hamburg itself is rather small and therefore a good starting point to see if the result is something I can use in our product. The conversion went well; it’s been quite fast (Macbook Pro, 2.9Ghz 6-Core i7, 32GB RAM). If Hamburg is quick, why not try the same with Germany? Well, it quickly turned out to take quite long. I was still optimistic about doing the same with Europe on my machine. Now the frustration began.After running it for 14 hours, it just crashed with an error, telling me there’s not enough space left. 700GB of free space is not enough? The raw source data is about 30GB. 700 GBs should be enough. Well, chances are high that I did something wrong. I reread the docs thoroughly to make sure I got everything right the second time. This time it crashed after about 12 hours with some non-descriptive error. I did the same steps as in the previous iteration. Great. Thanks for nothing.I thought about taking a different approach. As I only wanted to show flight paths and airports, I thought about getting rid of buildings and other details. Buildings are the biggest chunk in OSM data and therefore take up a lot of time and computing power.Luckily there are tools out there to help you edit raw OSM data. First, I started out using <a href="https://github.com/openstreetmap/osmosis/releases/tag/0.48.3">osmosis</a>. The latest release is from September 2020. Even though it’s not super up to date I still gave it a shot.While trying to figure out how to properly filter the raw data, I was able to shrink the size of the raw OSM data. Not extensively but at least a bit. However, even for Hamburg, it took quite some time.I reran the openmaptiles tools but still ran into numerous errors. OK, maybe there’s another tool to filter unwanted data? Say hi to <a href="https://github.com/osmcode/osmium-tool">Osmium</a>. It’s written in C++ and should be fast. I gave it a shot.I managed to also shrink the size of the OSM data. It is quite fast. Osmosis is nowhere near osmium, but for Europe, it still took forever, and at some point, I decided to kill the task. So, basically, I was back at square one.After doing some further research, I found <a href="https://github.com/systemed/tilemaker">tilemaker</a> as an alternative to openmaptiles. It’s also based on C++ and has no dependencies. Oh boy, fingers crossed. I started again with Hamburg. It went well. There it was — our MBTiles file.I dropped it into our tileserver-gl (which reads the MBTiles file and provides the requested tiles for the specific area the client needs) and voila I had a working map with self-converted and self-hosted tiles. Yeehaw! I couldn’t believe it. Mentally preparing for converting Europe, I reread the tilemaker doc just to get it right the first time. Unfortunately, the docs say tilemaker should only be used for small extracts and not whole continents.Good morning. It’s Groundhog day. Greetings to Bill Murray. WTF. I did some research and found <a href="https://wheregroup.com/blog/tilemaker-am-limit/">this blogpost</a> (german) by Marcel Normann.According to his findings, tilemaker uses quite a lot of RAM. He found out after doing some benchmarks, the factor is 12 RAM to filesize of the raw data.So, Europe would need around 256GB of RAM. This kind of RAM is not even affordable when outsourcing everything to AWS, so I nearly gave up. It can’t be that hard to just display a nicely styled map without relying on some third-party company.After taking some deep breaths and rethinking all the options, I went back to my initial approach, using the open source openmaptiles tooling with just a bit more resources.Luckily, AWS offers some powerful EC2 instances to make the process quite easy. This way it’s not blocking my machine the whole day because AWS offers compute time for reasonable prices.So below, you’ll find the solution which turned out to work quite nicely in the end — providing you’ve got enough computing resources.In the end, I learned a lot about OSM data itself, how to manipulate raw osm data, and how to trust your gut. Most of the time your intuition is right, and the first choice is sometimes the best. I had to learn it the hard way this time.<h3>The Way To Go</h3><ol><li>Launch an AWS EC2 Instance (I use an c5.4xlarge, which is optimised for computing tasks. It has 16 vCPUs, 32GB RAM, and a 500GB EBS gp2 SSD storage volume)</li><li>Install docker and docker-compose</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0059a24e1d6cf94ff91585bb2614562f/href">https://medium.com/media/0059a24e1d6cf94ff91585bb2614562f/href</a></iframe>3. Clone the required tooling for working with raw OSM data and convert it to a *.mbtiles file<iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f09a309dad9fbb43b1aaf17fce3bac1/href">https://medium.com/media/4f09a309dad9fbb43b1aaf17fce3bac1/href</a></iframe>Now this will take quite some time. Our last run took 48 hours for Europe.In order to provide the tiles to your frontend or mobile application, there are two options:Option 1: Run a server (e.g., a <a href="https://github.com/maptiler/tileserver-php">PHP-based tileserver</a> or <a href="https://github.com/maptiler/tileserver-gl">tileserver-gl</a>), which takes care of extracting the necessary tiles from the MBTiles file.Option 2: Extract the tiles and host them statically.I went with the latter approach as I always want to keep maintenance to a minimum, have an easy-to-grasp infrastructure, and leverage the caching mechanism of AWS Cloudfront.Using <a href="https://github.com/mapbox/mbutil">mbutil</a> is the way to go. It’s quite simple. Be aware that the t*.pbf files that are being extracted by mbutil are gzipped. So, if you use S3, make sure you include the flag —-content-encoding gzip when copying the files.This adds additional metadata to the object that is being stored in S3 and helps Cloudfront to handle it correctly. You can also un-gzip the files in case you want to gzip them on the fly or don’t want to use compression at all.<iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b34e2377483e025b6a9047290feb4f2b/href">https://medium.com/media/b34e2377483e025b6a9047290feb4f2b/href</a></iframe>Note: It took 48 hours to get an MBTiles file of Europe with an EC2 5c.4xlarge instance, with 500GB EBS gp2 Volume, running Ubuntu Server 20.04. I was constantly on edge, hoping for the process not to crash randomly in between.Optional step —Un-gzipping the files<iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b56bc225eecc124b6fb01581fbe8aeb7/href">https://medium.com/media/b56bc225eecc124b6fb01581fbe8aeb7/href</a></iframe>Optional step — Using a server to host and serve the tilesFor some reasons you might want to keep the tiles in one file (regular updates, for example) and leverage a tileserver to do all the heavy lifting for you. Serving the tiles, caching them, or even providing prerendered raster tiles are some benefits of running a tileserver yourself. It’s pretty straightforward to accomplish:<ul><li>Download/Upload T<a href="https://github.com/maptiler/tileserver-php">ileServer PHP</a> onto your server</li><li>Copy the *.mbtiles file in the root dir next to the other tileserver files</li><li>Open up your browser, navigate to your directory, and there it is.</li></ul>TileServer PHP also offers you a code snippet on how to render the map in a browser using <a href="https://github.com/mapbox/mapbox-gl-js">mapbox-gl-js</a> (as of mapbox-gl-js 2.x they’ve adjusted their licensing, and now you’re not allowed to use the js library without an active subscription anymore. Therefore you either have to use 1.x or <a href="https://github.com/maplibre/maplibre-gl-js">maplibre-gl-js</a>, for example).<h3>Tips</h3>1. In order to speed up the process, you can manipulate the raw OSM data with the <a href="https://osmcode.org/osmium-tool/">osmium-tool</a> by filtering out buildings (buildings take up a lot of space).2. Prevent using zoom levels beyond 14. Believe me, you don’t want to go there.3. If you plan on using smaller extracts e.g., single countries, I’d strongly suggest using <a href="https://github.com/systemed/tilemaker/">tilemaker</a>. In comparison to the openmaptiles approach it&#39;s insanely fast.<h3>Resources, Tools, Raw Data, and Other Useful Links</h3><ul><li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/compute-optimized-instances.html">AWS EC2 5c.4xlarge with 500GB EBS gp2 Volume, running Ubuntu Server 20.04</a></li><li><a href="https://github.com/openmaptiles/openmaptiles">Openmaptiles Tools</a> (Converting raw OSM data into MBTiles)</li><li><a href="https://osmcode.org/osmium-tool/">Osmium Tool</a> (Manipulating raw OSM data)</li><li><a href="https://github.com/maptiler/tileserver-php">TileServer PHP</a> (Serve MBTiles)</li><li><a href="https://download.geofabrik.de/europe.html">Geofabrik OSM Data</a> (Download raw OSM data)</li><li><a href="https://wheregroup.com/blog/tilemaker-am-limit/">Tilemaker am Limit</a> (Article about RAM usage of tilemaker. In German)</li><li><a href="https://openmaptiles.org/">Pre-Converted </a><a href="https://openmaptiles.org/">*.mbtiles</a> (quite pricy though for commercial use, depends on your budget)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd54e87f86a4" width="1" height="1" alt=""><hr><a href="https://betterprogramming.pub/you-dont-need-to-integrate-google-maps-or-mapbox-in-your-apps-cd54e87f86a4">You Don’t Need To Integrate Google Maps or Mapbox in Your Apps</a> was originally published in <a href="https://betterprogramming.pub">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Daniel Reiser

Hej, I am Daniel 👋

I am trying to solve problems through intelligent, elegant, and functional software.

Expertise & Skills

React & Next.js

Angular 2+

Amazon Web Services

Strong Fundamentals

Testing Frameworks

Accessibility

Infrastructure As Code

Some Articles

Rollbacks and Pull-Request Previews using CloudFront & lambda@edge without Any Downtime

Improve Your UX in Under 10 Minutes

You Don’t Need To Integrate Google Maps or Mapbox in Your Apps