I switched to Gmail aliases (+ sign). This brought some relief after applying filters to handle the aliases and mark emails as read or directly move them to trash. But this had its own limitations. For a human marketeer spammer seeing my email it was very clear what my actual email address was, the privacy of my email across multiple websites wasn’t really protected and some websites did not allow the + sign at all. I started mulling over hosting my own mail server with a catchall enabled so that I can create new mail address on the fly. But hosting your own mail server had its own perils and the internet was full of warning signs on why not to do this.

However, I had not given up and through the years I had a mental model of what sort of an app I needed. The biggest issue with hosting my own mail server was well-reputed IP addresses for outbound email and highly available servers for inbound email. I needed to use this app only to receive email notifications. With great trust in whatever the serverless fuss is about and an unused domain lying around I started building my own serverless email in 2018.

Building the Email Relay

I wanted to ensure I enjoy building the app while the project stays useful in the long term. I was also customer zero with some interest validated from a few friends. I had to be careful while building this app, as migrating from an email address can be painful so if this app is used by friends n’ family, it should be useful for a long time. I chalked down the following considerations.

1. Design the application for reliability: An email can arrive at any time and the sender MTA might not always dial in later.
2. Code changes should be shipped with least amount of effort: The fun of the project is to implement features. Ensuring they get shipped quickly accelerates the gratification of seeing a feature work or the despair of seeing it break. Making deployments easy was also going to cut down the pain during overnight bug fixes.
3. Spend near-zero time in maintenance: I do not enjoy maintenance neither have committed time for it. ZeroOps is essential.
4. Implement measures to prevent data loss: If me or my friends lose an email because the app failed to forward it, it should not be painful to recover the email.

Mail Server

Identifying a service that can help me quickly spin up a mail server capable of sending and receiving emails was crucial. I was familiar with SendGrid and had used it for outbound emails. To receive emails? not really and it seemed like that would need a lot of wiring of webhooks. I wanted to ensure that the service is able to trigger a forwarding logic seamlessly. This pushed me into looking for services with out-of-box integrations. AWS offered just that with Simple Email Service (SES), which could trigger Lambda functions. Perfect fit for building this quickly.

Relay

SES supports emails up to 10MB in size, so it is obvious that SES would only pass along metadata through its Lambda trigger. S3 worked as a good place to dump the complete email for the lambda to refer to and also acted as a backup for processed/unprocessed emails.

User Management

I also wanted to make this work for my friends and family. They should be able to sign-up and login to their individual accounts to manage the email aliases they have generated. The first problem to solve here, was managing user identities. I was certainly not going to build my own identity provider. AWS offered Cognito and it’s integration with Amplify seemed to be the way to go, but for whatever reason I found it more compelx against the comfort I had developed with Firebase. Remember? this had to be fun! So Firebase auth it is and naturally with frontend hosted on Firebase.

I was able to spin up a React based frontend with Cloud Functions to handle alias creation and management in the Firestore.

App Experience

Most of the signups I do are on my mobile device. There were also incidences where I had to write down my email for someone, and using an app to generate email addresses on my phone would be helpful. Building a mobile app is a taxing exercise. Fortunately, Progressive Web Apps exist. I ensured the React build has the right PWA manifest. So my users are able to download the web app and use it like a native app. WIN!

Architecture

With an array of pure serverless components, the email relay has been running gracefully for the last 5 years. The architecture looks like something like this. You might notice Simple Notification Service (SNS) which I haven’t mentioned earlier. SNS helped me split the email forward logic into two lambda functions, ensuring that the function triggered by the SES is able to acknowledge the SES trigger within the required 30 second timeout period. The SNS is used to pass on this event to the next function that does the heavy lifting to sending out an email which can run into intermittent failures.

Reflecting on last five years

When I built this, I optimised for three things: reliability, low maintenance, and fun. Five years in, it’s lived up to that.

The entire service has been costing me less than $2 per month to run for four active users. On average, it processes around 30 emails per day; not trivial traffic, but steady enough to prove it’s useful. There is something deeply satisfying about infrastructure that quietly does its job, month after month, charging your card only for the exact work it performs.

Maintenance has been almost nonexistent. In five years, I’ve spent under eight hours upgrading a deprecated Node.js runtime for one Lambda function. That’s it. No firefighting. No late-night ops panic. No scaling drama. It has largely embodied the ZeroOps principle I had set out to test. For a side project that touches something as sensitive and timing-critical as email, this adventure exceeded my expectations.

The Progressive Web App decision also turned out to be more important than I expected. Because it behaves like a native app, the handful of friends who use it actually formed a habit around it. When an app becomes muscle memory instead of friction, you know you got something right, not in terms of growth metrics, but in terms of the design intent.

Would I open it up to the public? Right now, I’m enjoying the quiet stability of it. Email is deceptively complex; deliverability, abuse vectors, spam compliance, privacy guarantees. The moment you open it broadly, you inherit a different class of responsibility. That’s a different project entirely.

Maybe someday.

For now, I’m content knowing that a small, well-designed serverless system that continues to hum along in the background, solving a very specific problem; reliably, cheaply, and almost invisibly. And sometimes, that’s exactly enough.

Before we look at these factors, let’s look into the background. Automation comes from the Greek word automatos meaning “acting on its own”, but when does anything ever act on its own? It begins with the intention to act in response to an event. This event forms the genesis of an action or a sequence of actions that resolve the intention. Some of these actions can run into exceptional scenarios and might need to be checked up on. Some of these exceptions might call for manual intervention. Some of these manual intervention if recurring, might need further automation. If there is doubt about the accuracy of the actions taken, a trace of the reasoning is needed to answer the how, what, when, and why’s. Analysis of which, might help decide further evolution in this automation.

Image Source - https://xkcd.com/1319/

Once we consider these factors in what we attempt to bring into being, we can support the lifecycle of a healthy and maintainable automation.

Cost

Acceptance of every initiative boils down to the value derived against the cost. It’s important to identify value and weigh it against the expenditure required for automation. Now, forecasting these benefits isn’t always easy. Sometimes, impossible. In such a case, careful observation of demand patterns can help base your decisions.

Prioritizing investment in the most valuable areas creates opportunity for quick validation. A drop in value against the cost indicates a need for improvement in the automation strategy. At the same time, a surge in demand (even at a later time) can signal an opportunity to expand the scope of automation.

Capitalizing on success while minimizing risks and resource allocation is the key. A phased approach to automation, influenced by changing value-cost ratios helps justify investments. In a finance department, automation could be applied to invoice processing. The value is derived from reduced manual data entry, fewer errors, and expedited payment cycles. The cost encompasses the development of automated workflows, integration with accounting software, and ongoing monitoring. The value can be gauged by tracking metrics such as processing time per invoice, error rates, and cash flow acceleration. If the value of faster payments and increased accuracy justifies the expenses associated with automation, the initiative becomes a wise investment.

Risks

Risks exist in forms other than cost. They extend to legal, ethical, and technical and relate to organizational domains. Along with technical feasibility, the rules and policies that apply to your industry and region are just as important. The risk is damage to reputation, legal compliance, and customer trust which outweighs any productivity gains.

Once you’ve assessed both cost and risk, the path forward becomes clearer. By prioritizing the reduction of risk, you gain clarity to build against a well-defined scope. This also ensures there isn’t a compromise on legal, ethical, or other organizational considerations.

Consider a scenario where a company aims to automate its CRM system to enhance customer interactions. While automating customer data management might streamline operations, there’s a risk of violating data privacy regulations like GDPR. If the automated solution inadvertently exposes sensitive customer information, the repercussions could be severe, including fines and reputational damage. In this case, the risk of non-compliance with data privacy regulations outweighs the potential gains from automation. The organization might need to descope certain functionalities or implement stringent security measures to ensure compliance and safeguard customer data.

Triggers

With the intention to act and the scope for action established, a trigger for action must also be decided. This is a balancing act and requires a deep understanding of the process, user behavior, and business goals. An inappropriately placed trigger can introduce bottlenecks or add excessive complexity. This greatly diminishes the value of the automation effort. Moreover, a trigger too aggressively placed might lead to significant development efforts. Eventually reducing the perceived value against the building cost.

It is important to consider the technical aspects. But that’s not all it takes to set up a perfect trigger. A keen awareness of user expectations and behavior is also necessary. Focus on keeping manual intervention and build effort low is key to ensuring maximized value.

In the IT domain, incident management automation intends to swiftly address and resolve system issues. The trigger for this automation should be real-time monitoring systems that detect anomalies or system failures. If the trigger is set to manual notification by end-users, critical incidents may go unaddressed until reported, resulting in downtime and customer dissatisfaction. Choosing the right trigger, such as automated monitoring tools, ensures that the system proactively identifies and addresses issues, minimizing downtime and maintaining service reliability.

Exceptions & Failures

Automation that runs complex, interdependent tasks is more likely to run into errors. Many of these errors will occur due to exceptional scenarios that need human attention. By identifying these scenarios ahead of time, you can create a comprehensive SOP. This prepares your maintenance team to address exceptions effectively. A well-prepared SOP can streamline manual interventions, ensure consistency, and reduce downtime. The SOP can also provide instructions for handling undocumented exceptions such as an escalation matrix. This maintains operational continuity even in unfamiliar situations.

It is also important to be able to restore automation after recovery. If a system failure causes an operation to halt, being able to resume the same operation after a fix is applied, minimizes business impact. Handling exceptional scenarios like these reinforces the credibility and value of automation.

In a hospital’s patient records automation, there might be instances where the system encounters incomplete or conflicting patient information. These exceptions can’t be resolved automatically and might necessitate human review. The SOP could guide the maintenance team on how to reconcile conflicting data, verify patient information, and ensure data accuracy. Additionally, intermittent errors like network interruptions might disrupt the automation temporarily. The ability to resume automation seamlessly after resolving the error prevents service disruptions and maintains workflow efficiency.

Notifications

It is common to run into situations where automation requires human attention. Could be for review of actions taken, or addressing errors that halt execution. Notification mechanisms in the form of emails, text messages, phone calls, or even automated alerts to external systems (such as webhooks) can be implemented. These notifications add value by reducing response and resolution time.

Moreover, mechanisms like webhooks and APIs create a dynamic ecosystem. Allowing new automation to interact with existing ones. This approach ensures that your automation efforts remain flexible, adaptive, and capable of accommodating future enhancements.

In essence, notifications connect automation’s technical efficiency with human decision-making. By implementing effective notification strategies, you enhance the impact of your automation. This aspect can be better understood through a concrete example.

For an automated financial transaction system, imagine an instance where a high-value transaction triggers multiple security alerts. The automation could generate an urgent phone call to the designated security analyst, highlighting the potential risk and prompting an immediate review. Simultaneously, if the system detects a failed transaction due to technical glitches, it might send an email notification to the IT support team, ensuring a swift resolution.

Skill to Operate

Some automation implementations need an operator. It could be to trigger the automation or assist the automation during operation. Users needing effort to develop proficiency to operate can discourage adoption. Scaling a team of operators is also difficult if the necessary skill levels are too high.

Striving to keep required skills as low as possible enhances the value generated by automation by improving impact. You can achieve this by offering user-friendly interfaces, comprehensive training, and clear documentation. This can ensure that automation doesn’t become a barrier for users with varying levels of expertise.

Imagine an analytics team using an automated script to generate monthly reports from a complex dataset. In situations where manual intervention is needed, such as fine-tuning parameters or choosing specific subsets of data, the level of technical expertise required can impact the automation’s value. To mitigate this, the team provides comprehensive training sessions and detailed documentation on script usage. Additionally, they create an intuitive web interface that simplifies parameter selection, transforming a potentially daunting task into a user-friendly experience. By ensuring that user intervention demands minimal technical skill, the automation’s value is magnified as more team members can confidently contribute to insights generation.

Scale

Automation consumes resources. A reality that becomes particularly obvious when it comes to scalability. An increase in demand can push automation to its limits. When automation can’t scale gracefully, it risks degrading performance, causing frustration, and reducing the value it’s intended to provide.

To ensure that value remains high, it’s essential to align resource allocation with demand as closely as possible. This involves leveraging serverless platforms that automatically allocate resources based on real-time needs. Alternatively, using queue-based systems can prevent resource overload by controlling the rate at which automation processes incoming requests.

Imagine a customer service automation utilizing chatbots to address customer queries. During peak hours, if numerous customers initiate chat conversations simultaneously, the system might struggle to provide prompt responses. A scalable approach could involve routing chat requests through a queue system. This not only prevents resource overload but also ensures that each customer receives timely assistance, maintaining customer satisfaction even during high-demand periods.

Logs & Metrics

Analyzing the performance of the automation, whether functionally or technically, is also important. Logs that capture the context of inputs, decisions, and system states provide a clear narrative of the automation’s actions. This helps establish trust in the outcome.

Moreover, the value of automation remains theoretical unless backed by concrete usage metrics. Capturing data such as demand patterns, turnaround times, and operational costs helps validate the automation’s effectiveness. They also unearth insights for continuous improvement. These metrics can reveal areas where the automation excels and pinpoint potential bottlenecks or areas for enhancement, guiding strategic decisions on future investments.

In the financial sector, an automated investment recommendation system might suggest portfolio adjustments to clients. Without a transparent record of the data sources, algorithms, and market trends considered in generating these recommendations, clients might be skeptical of following the advice. By maintaining comprehensive logs that trace the journey from data inputs to output recommendations, the automation bolsters its credibility. Additionally, tracking metrics such as the success rate of recommendations, time taken for portfolio adjustments, and cost savings due to reduced manual analysis reaffirms the value of automation.

Evolution

Every automation will show room for improvement. Either by optimizing the approach or extending the scope. Keeping the implementation flexible for changes/updates supports such improvements. A rigid implementation that lacks room for extension or adaptation limits its ability to evolve. On the other hand, a flexible architecture enables low-cost experimentation and continuous improvement.

By allowing the introduction of new systems, data sources, or behavioral changes you support innovation. The result is an automation strategy that retains its value against evolving needs.

Imagine a marketing team utilizing an automation platform to send out email campaigns. Initially, the automation ensures emails are sent at optimal times based on general engagement data. However, upon analyzing metrics, the team realizes that certain segments of customers exhibit higher engagement rates during specific days of the week.

To capitalize on this insight, they extend the automation by incorporating customer segment-specific scheduling. Now, emails are tailored to each segment’s engagement patterns, resulting in increased open rates and click-through rates. By allowing for this extension based on metrics analysis, the automation adapts to customer behaviors, significantly boosting campaign effectiveness and overall value.

Conclusion

To summarize,

• Balancing value against costs ensures efficient allocation of resources, allowing for phased automation investments.
• Mitigating risks related to legal, ethical, and technical aspects safeguards automation value.
• Well-placed triggers align with user behavior, optimizing automation’s value while minimizing development effort.
• Managing exceptions and intermittent errors through comprehensive SOPs reinforces automation’s reliability and value.
• Effective notifications bridge technical efficiency with human decision-making, enhancing automation’s impact.
• Minimizing required operator skills through user-friendly interfaces eases adoption.
• Scaling gracefully with demand ensures automation continues to deliver value without degrading performance.
• Comprehensive logs and metrics build trust and provide data for continuous improvement of automation.
• Flexibility for changes and extensions supports ongoing improvements and retains automation’s value.

Automation is more than just a mechanism to transfer data or calculate solutions. With the right implementation a dull, lethargic process can turn into a transformative force. By listing these factors, I hope to help you as an engineer or a decision maker to boost your processes with automation. If you have feedback please leave a comment or drop a message to om [at] 0x8 . in.

Further reading

• Simple Systems have less downtime
• The hardest part of building software is not coding, it’s requirements
• Entropy in Engineering Processes

Services like image/video processing apps can be memory intensive, requiring autoscale configurations to be backed by observing memory usage metrics. Calculating resulting average metrics can be a challenge when relying on memory. A detailed discussion on this challenge is available here https://medium.com/@jonfinerty/flapping-and-anti-flapping-dcba5ba92a05.

To summarize - When running an application on a Virtual Machine or Azure App Service plan, the memory metrics account for application and operating system usage combined. When scaling out, fresh instances of the application are added, which also add operating system footprints. When average memory usage across all instances starts receding below scale-in thresholds, autoscale tries to predict the resulting metric (average memory utilization) based on the distribution of current metric. This prediction captures a margin of error due to the operating system footprint belonging to excess instances which will be wiped out after a scale-in operation. This error triggers the anti-flapping mechanisms, delaying the scale-in operation.

Scaling in

We have the following model from the previous post that calculates the resulting metrics during scale-in operations.

r_i = (x * m) / (x - s)

Here, x represents the allocated instances, m represents the memory utilization at any given time, s represents the number instances being removed during scale-in and r_i represents the resulting scale-in metrics.

In this calculation, m is the instantaneous memory usage, which includes the operation system footprint. Memory used by instances being scaled in (removed) should be discarded from this calculation to get a better approximation of the resulting memory metric.

It is reasonable for autoscale to face challenges while differentiating operating system and application memory. Therefore the onus is on us to estimate the operating system’s memory footprint. This can help us identify system states (allocated instances, scale-out, scale-in thresholds, etc.) when scale-in/out can be challenging. We modify the model above to account for the memory being used by the operating system in percentage (b) and exclude the portion being used by the instances to be scaled-in.

r_i = [(x * m) - (b * s)]/ (x - s)

The plot (black) below shows the resulting memory usage at allocated instances along X-axis using the model above. Take note of the scale-out threshold (red) at 60%, scale-in threshold (green) at 50% and the current memory utilization (purple) at 49%. Number of instances to scale-in is set to 2. You can click on the Desmos link and modify the values using sliders to understand the relationships with the resulting metric. The shaded area indicates allocated instances that will experience flapping at given m.

The plot (blue dotted) indicates the actual resulting metrics after excluding the operating system memory usage from scaled-in instances. This reference helps understand the margin of error in calculating resulting metrics.

From the plot, we can observe

• At 6 instances, if memory usage drops to 49% autoscale will calculate the resulting memory usage as 73%. This will block the scale-in operation as autoscale predicts a scale-out operation.
• For scale-in to occur at 6 instances, current memory utilization will have to drop to 39% (try sliding m to 39%).
• Though there is an error in the resulting metrics at 11 instances and above, both metrics lie below the scale-out threshold indicating that the above configuration is safe to use for applications which normally operate on 11+ instances with average memory utilization at 50%.

Scaling out

The above effect on average metric calculations is observed during scale-out too. We apply the following model to calculate resulting scale-out metric (r_o).

r_o = (x * m) / (x + s)

This again has margin of error as the resulting metric does not include the additional memory usage added by the scaled out instances (s). Let’s repeat the process to factor that in considering memory being used by the operating system in percentage (b).

r_o = [(x * m) + (b * s)] / (x + s)

The plot below is identical to the one above designed to indicate resulting scale-out metrics. The only difference being, the shaded area highlights allocated instances for given m where the actual memory utilization due to operating system overheads is bound to cause further scale-out operations.

We can observe,

• If memory utilization spikes to 65% when allocated instances are 15 or more, multiple scale-out events are bound to happen in a short duration to cover for the operating system overheads added by new instances.
• The autoscale configuration is safe to use for services that will not exceed 15 instances.

While multiple scaling events or undertaking excess resources is normal for autoscale operations, the plots above help identify the limits within which our configurations will operate optimally. We can identify the normal operating range of instances, estimated operating system memory usage and then identify the scale-in and scale-out thresolds such that the resulting metrics calculated by autoscale and actual metrics are as close to each other as possible.

Ideas presented here are based on my personal observations. Please maintain caution when applying configurations on your own cloud environments. Your results might vary. Got feedback or ideas? Drop a comment or email om [at] 0x8 dot in.

Scale-outs are usually simple. You define an upper bound on your metric and autoscale will take care of the rest. Anti-flapping mechanisms with prevent unsafe scale-ins from happening so it is scale-ins that should be better estimated. An incorrect configuration can lead to unnecessary costs. You can be a lot more confident about your autoscale configurations, if you know precisely when a scale-in is going to occur keeping in mind all the factors that prevent a scale-in from happening.

We have identified the following model that helps calculate the resulting scale-in metric (r_i) based on the applied scale-in, scale-out thresholds (t_i, t_o respectively) and scale-in instance count (s). We consider t_i as this is the metric where the scale-in intent is triggered (view plots in previous post).

r_i = (x * t_i) / (x - s)

A more generalized form of the above equation would be

r_i = (x * m) / (x - s)

Where m is the metric upon which autoscale is being configured.

Flapping will occur when the metric falls below scale-in threshold t_i and the resulting scale-in metric r_i is greater than scale-out threshold t_o.

r_i ≥ t_o when m ≤ t_i

However, this doesn’t block scale-in. Only delays it until the metric falls further. This takes us to the important question. For the allocated instances, what value should the metric reach to definitely cause a scale-in?

For the allocated instances, what value should the metric reach to definitely cause a scale-in?

We need to find a metric value m for a given instance count x, scale-in instances s which upon scale-in leads to a resulting scale-in metric r_i that is just slightly below scale-out threshold t_o i.e. t_o - 1.

Let’s build on what we have established for r_i

(x * m) / (x - s) = t_o - 1
∴ m = [(t_o - 1) * (x - s)] / x

For any given x, t_o and s, we are now able to calculate the effective scale-in threshold. Take note, that scale-in threshold t_i is still important to consider as for any value of m that falls above t_i, the effective scale-in threshold will always be t_i.

m = [(t_o - 1) * (x - s)] / x where { m < t_i }

Let us plot a graph for m (orange). This indicates the effective scale-in threshold. i.e. Scale-in will occur only if the metric falls below this value for given system state. The shaded area indicates the region where m will cause flapping. For optimal autoscale configurations, instance count x during average operating hours should not fall under the shaded area.

Some observations that can be made

• Flapping will not occur for scale-ins beyond 10 instances.
• At 6 instances, the metric will have to fall below 74% for scale-in to occur

We can now identify if this meets our normal operating ranges or optimize the autoscale configuration to ensure that effective scale-in thresholds are met.

Feel free to open the Desmos link and play around with the parameters configured to explore further.

In the next part we will dive into challenges faced when using memory based metrics for autoscale configurations.

Ideas presented here are based on my personal observations. Please maintain caution when applying configurations on your own cloud environments. Your results might vary. Got feedback or ideas? Drop a comment or email om [at] 0x8 dot in.

1. Understanding flapping (current)
2. Identifying effective scale-in metrics
3. Challenges with memory based metrics

The unpredictability of compute or storage demands of web applications made the Cloud happen. The ability to scale as per your instantaneous requirements and being charged just for those instances is the crux of the pitch for moving to the cloud. This what the autoscaling capabilities of Azure Web Apps help you achieve. Autoscale adds more compute instances (scale-out) or increase the size of your existing instance (scale-up) based on metrics that you define. While scaling out is important for the availability of your service, scaling in is important to keep costs under control.

For example - you may choose to add 1 new instance to your App Service Plan if the CPU usage remains above 90% consumption for more than 15 minutes. While scaling up is great, it is also important for your App Service Plan to scale down once peak traffic is over and the stress on your application starts receding. So you can choose to reduce 1 instance from your App Service Plan if the average CPU usage across all instances reduces below 80%.

This summarizes the autoscaling capability; however, the behaviour of the metrics and thresholds that define your autoscale rules can be slightly nuanced. Let’s build on top of the previous example.

• Assume that the application started receiving high volume of requests at 6:00am continuing to increase average CPU usage to 90% until 6:30am. This will trigger the scale-out rule a couple times, adding 2 instances to the App Service Plan making the total instance count 4.
• The traffic is now shared by 4 instances and average CPU usage drops to 80% while the same volume of traffic continues.
• At 7:30am traffic gradually starts receding and average CPU usage drops to 79% eventually triggering the scale-in rule reducing the instance count to 3.

While this might seem expected, let’s understand implications here. A total of 4 instances where signalling average CPU usage of 79%. Reducing a single instance with existing traffic will lead to (4 x 79) / (4 - 1) = 105.33% average CPU usage. This is far less than ideal behaviour and will eventually create a scale-in, scale-out loop as seen below starting 7:30 or even worse, crash the application servers.

Azure understands that this will lead to unhealthy application performance and therefore implements anti-flapping measures. i.e. before performing any autoscale operation, Azure predicts if the operation will trigger other contradicting operations. If the answer is yes, the operation is cancelled and you will end up seeing “Flapping” in your autoscale operation logs. The autoscale operation in practice looks as below.

How do you prevent flapping?

While anti-flapping measures on Azure protect you, it is a good practice to take calculated steps to keep these events predictable and rare.

In the above example we have default 2 instances and scale instance is 1. Which is essentially adding 50% more compute. At peak operating point, there are 4 allocated instances. Scale-in of 1 instance causes 25% drop in compute capacity. Flapping would not be observed under the same autoscale configuration above if the peak number of allocated instances was 10. Meaning the configurations is suitable if the services normally operates with 10 or more allocated instances. Most common occurrence of flapping is found when the ratio of scale instances to operating instances is high or scale-in and scale-out thresholds are too close. A mathematical model can help paint a clear picture.

We consider the number of instances being added or removed in a single autoscale operation as Scale instances (s), the number of instances being used at any point in time as allocated instances (x) and scale-in scale-out thresholds (t_i, t_o respectively).

In the graphs below, the X-axis indicates allocated instances while the Y-axis indicates the autoscale metric (average CPU% in this case). The green line is the scale-in threshold while the red line is the scale-out threshold.

Scaling in

The plot (blue) uses the following equation to calculate the resulting scale-in metric (r_i) considering the current metric has reached the scale-in threshold (t_i).

r_i = (x * t_i) / (x - s)

Autoscale will operate a scale-in if the metric falls below 80%. Flapping can be observed for all scale-in operations when the allocated instances are less than 9. In this range the metric shoots above the 90% scale-out threshold which will trigger a scale-out immediately. Therefore, Azure will prevent the scale-in operation with its anti-flapping measures. This does not mean scale-in does occur at all. It simply means that the metric will have to fall lower for scale-in to actually take effect.

Scale out

More instances don’t hurt an application so scale-out will work as per thresholds. Autoscale does not block scale-out operations as part of the anti-flapping mechanism. The costs still apply so it is at least insightful to apply the math for scale-out too.

The plot (purple) uses the following equation to calculate the resulting scale-out metric (r_o) considering the current metric has reached the scale-out threshold (t_o).

r_o = (x * t_o) / (x + s)

It can be observed that scale-out works optimally for all scale-out operations above 8 allocated instances.

Tune it

With the model in front of us, we can now tweak our autoscale settings considering the traffic patterns on our services. Feel free to click on the Desmos link on the graph below to play around with the autoscale configurations to find a sweet spot for your autoscale configurations.

In the next part we will discuss an approach to identify effective scale-in metrics.

Ideas presented here are based on my personal observations. Please maintain caution when applying configurations on your own cloud environments. Your results might vary. Got feedback or ideas? drop a comment or email om [at] 0x8 dot in.

When running web applications at scale, load-balancers act as gateways to a cluster of homogeneous servers creating a reliable infrastructure. These load-balancers can route traffic to servers using algorithms as simple as round-robin or get complex and probe into server health metrics to choose the most reliable unit. Conventionally, load-balancers operate on layer 4 traffic peeking into TCP and UDP packets and routing them to the optimal destination. When speaking in context of Azure, it would be the Azure Load Balancer. Load-balancing does get a bit more nuanced once you think about scaling globally. This post will walk you through these nuances and help you appreciate the role that the DNS protocol plays here.

The need for global load-balancing

Consider an application like OneDrive that provides personal storage space to users across the globe. Running an application of this nature at a global scale needs more than just a single server cluster. An approach that accommodates geographically distributed clusters becomes essential along with the ability to process user requests at the topologically nearest cluster. This approach will ensure requests sizing up gigabits of data will spend less time on the network.

Conventional load-balancers act like a gateway and face all the traffic flowing towards the cluster. Geographical placement of the cluster behind a load-balancer is inconsequential as the topological distance on the network will vary only behind the load-balancer. This approach does not process requests over an optimal network route. A more appropriate solution would let users discover their topologically nearest cluster and process requests locally.

Clients looking to send a request to OneDrive should be able to discover the nearest cluster before the actual request is initiated. This is where DNS plays a interesting role.

DNS Background

DNS servers help clients such as web browsers, mobile apps, etc. translate domain names to IP addresses where the actual HTTP request is routed. I will reiterate through the relevant DNS concepts in this post. Cloudflare has a nice article providing an overview if you are looking for more - what a DNS is and how it works.

There are many types of DNS servers of which the Recursive Resolver and Authoritative Name Server are relevant here. A Recursive Resolver is a DNS server that acts as a concierge for client applications. This server does not always hold the translation of the domain name to the IP address, but is capable of traversing through a series of DNS servers and reach the Authoritative Name Server which holds this translation.

When a DNS resolver starts looking for an IP address hosting a domain name, it may come across records redirecting it to other Name Servers (NS record) or Domains (CNAME record); but the search will always end with an IP address (A record) queried from an Authoritative Name Server. This result is then cached based on the Time-to-live (TTL) value associated with the record.

(RFC1034, RFC1035) published in 1987 formalized these key characteristics of the DNS protocol.

Load-balancing with DNS and challenges

DNS being the mediator, introducing clients to their destination IP address; can be leveraged to balance traffic being routed towards a list of IP addresses where the web application is accessible. For example - multiple A records are supported. Resolvers upon receiving Resource Records (RRs) for a Domain Name containing multiple A records can prioritize an IP address based on various algorithms. So it isn’t wrong to say that DNS inherently supports load-balancing.

Dynamically formed Resource Records

Web application administrators have little to no control on the load-balancing behaviour carried out by resolvers. Resolvers are selected by the clients and are usually owned by the ISP or a service that the client subscribes to. Furthermore, load-balancing via A records is restrictive in context of global scale. The ability to configure a regional domain alias (us.onedrive.com, in.onedrive.com, etc.) or regional name server is desirable due to the flexibility of procuring sub-domains as opposed to IP addresses.

Conventionally, Authoritative Name Servers are designed to respond to every query against a domain name with the same Resource Records. A global load-balancing control plane on a DNS server can only be achieved through Authoritative Name Servers generating algorithmic responses. For instance, an Authoritative Name Server can be programmed to respond with Resource Records based on the topological origin of the request. The responses continue to comply with established standards, but vary based on the origin of the request.

A global load-balancing control plane on a DNS server can only be achieved through Authoritative Name Servers generating algorithmic responses.

This is precisely how services like Azure Traffic Manager work. There are numerous approaches to load-balance using DNS. Here are few of the many approaches supported by Azure Traffic Manager.

• Geographic: Select Geographic routing to direct users to specific endpoints (Azure, External, or Nested) based on where their DNS queries originate from geographically. With this routing method, it enables you to be in compliance with scenarios such as data sovereignty mandates, localization of content & user experience and measuring traffic from different regions.
• Multivalue: Select MultiValue for Traffic Manager profiles that can only have IPv4/IPv6 addresses as endpoints. When a query is received for this profile, all healthy endpoints are returned.
• Subnet: Select Subnet traffic-routing method to map sets of end-user IP address ranges to a specific endpoint. When a request is received, the endpoint returned will be the one mapped for that request’s source IP address.

A full list of DNS routing algorithms supported by Azure Traffic Manager can be found here - Traffic Manager routing methods.

Origin of the DNS Query

If you have paid close attention, you will notice that the query for Resource Records is made by an intermediate Resolver chosen by the client. The topological position of the Recursive Resolver is often close to the client, but users can choose/configure a Resolver that resides in a different country or continent for a variety of reasons. This leads the Authoritative Name Server in serving inappropriate Resource Records based on the Resolver’s location.

Since 2016, the DNS protocol supports an Extension Mechanism for DNS or EDNS0 (RFC6891) that is popularly used by intermediate Resolvers to add attach client subnet information to a DNS query. Such an attachment is called EDNS Client Subnet or ECS (RFC7871). This extension provides Authoritative Name Servers with the client subnet details from where the request originated and respond in context of this information while ignoring the address of the intermediate Resolver.

The effect being that the same Resolver will yeild different Resource Records based on the client’s subnet.

Bringing the DNS closer

With the implemention discussed above, Authoritative Name Server will still be accessible over a distant network for majority of global consumers. Most applications can deal with this level of inefficiency considering that the intermediate Resolver and the Authoritative Name Server share a few hundred bytes of data that eventually gets cached.

Authoritative Name Servers residing in the same region create a single point of failure in event of a disaster. Applications that wish to implement multiple failover regions in event of a regional blackout, value an optimized route to the Authoritative Name Server. Such applications find utility in implementing their Authoritative Name Servers over an Anycast network. Anycast is a routing method that allocates a single IP address to multiple servers. Client sending requests to the IP address usually end up connecting to the nearest server accessible on the IP address. Learn more about Anycast here.

So why not utilize Anycast for the server cluster itself? This is a valid use case, however Name Servers over Anycast are popular as they make it is easier to spin up a new cluster and route traffic to it by modifying records on a Name Server accessible over Anycast. Adding new clusters over an Anycast enabled IP address involves creating a new Anycast node, which is significantly more effort.

Load-balancing in action with Azure Traffic Manager

Azure Traffic Manager implements DNS based load-balancing using the techniques described above. Let’s take a look at it in action.

I have created a Azure Traffic Manager resource and configured the routing method as “Geographic”. There are 2 endpoints registered that look like this.

The result for DNS queries from clients in Pune, India and New York, US for the traffic manager domain name leads to the following results.

QUESTION SECTION:
omklabtest.trafficmanager.net. IN      A

ANSWER SECTION:
omklabtest.trafficmanager.net. 0 IN     CNAME   omktestapp-in.azurewebsites.net.
omktestapp-in.azurewebsites.net. 0 IN   CNAME   hosts.omktestapp-in.azurewebsites.net.
hosts.omktestapp-in.azurewebsites.net. 0 IN A   52.136.50.1

Pune, INDIA (routed to South Asia datacenter)

QUESTION SECTION:
omklabtest.trafficmanager.net. IN      A

ANSWER SECTION:
omklabtest.trafficmanager.net. 0 IN     CNAME   omktestapp-us.azurewebsites.net.
omktestapp-us.azurewebsites.net. 0 IN   CNAME   hosts.omktestapp-us.azurewebsites.net.
hosts.omktestapp-us.azurewebsites.net. 0 IN A   20.49.104.31

New York, US (routed to East US datacenter)

At the time of writing the Azure Traffic Manager costs $0.54/million DNS queries with additional costs for health checks peaking at $2/endpoint/month. If your web app is being served via 6 continents with over a million daily DNS queries, you pay approx ($2 x 6) + ($0.54 x 30) = $28.2/month. Check the pricing page for more details - Azure Traffic Manager pricing.

This summarizes what goes under the hood of services like Azure Traffic Manager and how DNS based load-balancing can help you build a global scale web application.

Have thoughts or questions? Drop a comment or email me at om[at]0x8.in.

Why go through a serverless approach at all? File uploads are tricky business and in the era of serverless solutions, I see little reason to route heavy HTTP transactions through the primary application infrastructure, especially when dealing with large file uploads. Anything above 5MB should not be routed to your primary application host for reasons documented at the end of this post.

This is what a video processing application might look like with serverless file uploads.

Channel 9 Studio could only do justice to the core concept in 4 minutes. In this post, I’ll document a few important considerations.

Before you take this to production

You may have noticed already that the crux of this approach lies in off-loading the complete logistics of a file upload to the storage account. You need to consider the following.

User backed SAS Keys are irrevocable

SAS keys once handed over to your clients (web, mobile, etc.) are irrevocable. If the SAS key isn’t generated correctly, your storage account is at risk of being exposed to abuse.

File validation

The storage account does not validate the uploaded file. So the container where the file is being uploaded should be assumed unsafe by all other parts of your application. I would in fact recommend maintaining a dedicate storage account and container for uploading files. These files can be moved within the perimeters of your application once validated by a logic app or a blob triggered function.

Note: Azure Advanced Thread Protection supports scanning of storage account blobs for possible malware using Hash Reputation Analysis. read more

File size limits

Storage account does not limit the size of files being uploaded. The maximum size of the file is dictated by the blob type. Keeping a short expiry on the SAS key doesn’t help either.

I tried creating a key that expires in 50 seconds and was able to upload a 1 GB file in ~9 minutes and 2GB file in ~17minutes through 2 separate tests. This likely because the HTTP transaction started within a valid timeframe.

Way forward

Does this mean that you cannot off-load file uploaded to a storage account? Certainly not. You can very well identify the risks and place mitigation measures.

Do not generate SAS keys for unauthenticated users

Allowing file uploads by anonymous users should be a strict no. You leave little opportunity to control abuse of a resource that you are getting billed for in terms of storage and bandwidth.

Generate a well scoped SAS token

Ensure that your SAS key is narrowly scoped. For a file upload use case you can ensure the following

• Start expiry few seconds before current time to prevent clock skew issues
• Keep end expiry reasonably short to prevent abuse and just long enough to support slow connections. No harm in keeping it short and increasing the duration based on user feedback.
• Provide “create” permission only. Users should NOT be able to read the file they uploaded. This opens more opportunity for abuse.
• Scope the SAS key to container and file name. Ensure that the file name stays unique for all users/sessions.
• Include requestor’s IP address in the SAS key scope to prevent abuse from other networks.

BlobSasBuilder sasBuilder = new BlobSasBuilder()
{
	BlobContainerName = ContainerName,
	BlobName = BlobFileName,
	Resource = "b",
	StartsOn = DateTimeOffset.UtcNow.AddSeconds(-10),
	ExpiresOn = DateTimeOffset.UtcNow.AddSeconds(120),
	IPRange = new SasIPRange(ClientIPAddress)
};

sasBuilder.SetPermissions(BlobSasPermissions.Create);

Read more about SAS best practices here.

Maintain logs of file upload sessions

Maintain logs that can help trace back a file to the user and possibly the IP address that uploaded it. In event of abuse these identifiers help you block future transactions from the same source.

Use blob triggered Function/Logic App to validate and move files to you application perimeters

Your implementation could be supporting a simple photo upload for a user’s display picture or large file uploads for a video processing/hosting service. You can implement a blob triggered function that validates the file’s size and content before moving it to primary application perimeters for consumption.

This is what the video processing application may look like with a few more measures in place.

But why not use my existing infrastructure?

It maybe a few lines code that extend your existing application to allow file uploads, but this story isn’t about code.

Memory

Depending on your file upload implementation, the file being uploaded is going to reside completely or partially in your application memory. This directly affects the scalability of your application. Especially concerning if the file upload behaviours are inconsistent and difficult to plan ahead.

Sockets

HTTP requests consume socket connections. Sockets, just like Memory or CPU are exhaustible. Large file uploads hold up sockets for longer durations impacting your server’s ability to serve other clients. There are slow networks that can upload files at speeds lower than an megabyte per minute. Such clients will hog your connection pool over long durations. This can also act as an attack vector for DoS attacks.

Security

File upload endpoints also open opportunity for buffer overflow attacks. Malicious files can only be validated once they reach your servers. This is risky especially if the same servers are being used to process sensitive information for other customers.

Do you think there more that needs attention when exposing a blob storage to act as a broker for file uploads? Drop you thoughts and suggestions in comments. Cheers.

Some basics

The title already tells you that there are two claims in a JWT token that we are most interested in. For those who haven’t had their essential visit to the Microsoft Access Token or ID Token documentation - UPN, Email or Phone numbers aren’t guaranteed to be unique system wide. UPN (User Principal Name) especially misleads devs into assuming uniqueness.

The only two eligible candidates for guaranting unique values within a tenant are OID and SUB.

OID vs SUB

This is what you are here for. Both of these claims guarantee uniqueness. There are subtle differences like the degree of uniqueness that these values provide.

Object Identifier Claim (OID)

In context of Microsoft Identity Platform this property maps directly to Object Identifier in the directory. For Enterprise applications integrating with Microsoft services, this becomes the easy choice. This property guarantees uniqueness in general. However, pairing the OID with TID (Tenant ID) provides additional defence against an accidental GUID collision.

By design, for the same user on different tenants (as a guest user), applications will get a different identifier. This is to protect a user’s directory or application information on a tenant from being mapped to another tenant.

However, a user from the same tenant appearing on multiple applications will share that same OID. This has prompted Microsoft to enforce apps to have the profile scope enabled to obtain this claim for users.

It is also important to note that OID does not find place in the JWT RFC (rfc7519). So if your application accepts tokens from multiple Identity Providers, OID might NOT be your best choice.

Subject Claim (SUB)

The subject claim is guaranteed to be unique within a tenant and within the application as well. Meaning that the same user appearing on two different applications will use a different SUB value. This makes it difficult for applications to map a user’s data across two different applications. This is highly advantageous for applications that would like to meet high privacy standards.

Additionaly, SUB is supported via the JWT token RFC (rfc7519). Meaning, every Identity Provider passing on JWT tokens is bound to provide a SUB claim that is populated with a unique value per user. If you are looking at extending support to multiple Identity Providers, SUB would be your best choice.

TL;DR

• OID works best for applications that are looking to integrate exclusively with the Microsoft Identity Platform.
• OID works best if applications rely on the directory identifier to integrate with other applications.
• SUB works best for applications aiming to serve high privacy standards.
• SUB works best for applications that are looking to integrate with multiple identity providers serving fully complaint JWTs.

There are proven benefits of abstracting out servers for certain solutions where the focus lies more on the business logic and lesser on the logistics of it.^[1][2][3] Serverless offerings are modelled for simplicity, but this can quickly lead to limitations especially while building enterprise focused solutions surrounded by niche requirements or compliances. Azure seems to have clearly inherited Microsoft’s proficiency in catering to Enterprise needs. Azure’s serverless offerings have managed to extend an array of features that help build compliant solutions with fair degree of ease. To evaluate just that, here we look at an Enterprise Integration problem and design a solution using Azure’s serverless offerings and highlight some of their capabilities.

Problem statement

North Wind Corp. runs an ERP solution where Request for Proposals (RFP) are uploaded for Vendors to access. North Wind wants to send out notifications to suppliers when they intend to raise RFPs. Vendors like Proseware Inc. would like to stay up to date on RFP events and trigger their internal business processes.

For the sake of simplicity, let’s break this problem statement down to three phases and address each of them with one of Azure’s serverless offerings.

Separation of Concerns

This is a cross-enterprise integration scenario. It is therefore important to understand that the solution needs to have clear separation of concerns between the two parties, North Wind and Proseware.

• North Wind’s sole responsibility should be to issue the RFP notifications to all vendors along with relevant metadata.
• Proseware and other vendors should be able to manage and route their own subscriptions.

Solution

North Wind is required to set up a notification infrastructure that its vendors can subscribe to. It is crucial for this infrastructure to support access for vendors to manage their own subscriptions.

RFP Notification being an event, Azure’s Event Grid starts to seem like a good candidate. Event Grid is designed to be used in event-based architectures where applications can emit events. Other applications and services can then subscribe to events of their interest. The general Event Hub offering supports deep integration with Azure allowing your Azure resources to readily start emitting event on the grid. The support for filters is also a huge plus especially for applications that would like to focus on a very specific range of events. If you are interested in a detailed overview, I would encourage you to read up if their Azure documentation.

If you want to improve your understanding of the key differences between Event Hubs, Event Grid and in some cases Service Bus head here https://docs.microsoft.com/en-us/azure/event-grid/compare-messaging-services.^[4]

We can approach a design where we create a topic dedicated for RFP notifications, but RBAC on the topic would not control vendors from accessing each other’s topic subscriptions. We can create a custom topic for each vendor allowing them to control all subscriptions on that topic. As each topic will have its own event publishing endpoint, North Wind’s internal ERP system will have to emit notifications on each topic endpoint. There should be a better approach.

All of the above constrains can be gracefully bypassed with Event Domains. Extending the same Event Grid infrastructure, domains allow applications to emit events across multiple topics over the same endpoint and at the same time allow RBAC to be applied on each topic created under the domain^[5]. We can therefore have a dedicated topic for each vendor under the Event Domain and assign the EventGrid EventSubscription Contributor role to Integration teams at vendors like Proseware. Note that this role is not visible in the Azure Portal at the time of writing. This role can be assigned via PowerShell.

Source: Example of topic based access from Azure Documentation - Event Domains

There is still one last caveat here. The event size on Event Grid is limited to 1 MB (messages are metered at 64 KB)^[6]. There can be RFPs larger than 1 MB and contain multimedia which is not suitable for the JSON schema supported by Event Grid. Even from a design stand point, the Event object should contain only enough metadata for the vendor to be able to locate the actual RFP and consume it into their processes. The location for the actual RFP is trivial to the problem statement as it could very well be in the ERP itself with authorized vendors allowed to pull information. For the sake of brevity, we opt for a Blob storage. We utilize the data property in the Event Grid schema to populate RFP metadata.

{
    "topic": "proseware",
    "subject": "rfp/new",
    "id": "11568938",
    "eventType": "new",
    "eventTime": "2020-10-02T13:41:00.9584103Z",
    "data":{
        "id": "IN934NZ12",
        "xml": "https://northwind.blob.core.windows.net/trader-rfps/rfp.xml"
    },
    "dataVersion": "",
    "metadataVersion": "1"
}

The RFP XML document hosted on the North Wind blob storage is structured as follows.

<root>
    <title>Intent to create the largest Lego Death Star</title>
    <id>IN934NZ11</id>
    <type>M582</type>
    <orders>
        <order>
            <name>Darth Vader</name>
            <quantity>1</quantity>
        </order>
        <order>
            <name>Storm troopers</name>
            <quantity>60</quantity>
        </order>
        <order>
            <name>Emperor's Royal Guards</name>
            <quantity>20</quantity>
        </order>
        <order>
            <name>Square blocks</name>
            <quantity>80000000</quantity>
        </order>
        <order>
            <name>Blasters</name>
            <quantity>176</quantity>
        </order>
    </orders>
    <note>Vendor to take care of shipping and other logistics.</note>
</root>

The solution at this point looks something like this. We have everything in place to:

1. Emit events to a single endpoint.
2. Provide isolated control for each vendor’s Active Directory backed guest users to setup event subscriptions on their own topics.

Separation of Concerns between North Wind and Proseware.

Integration Point

Proseware, like most businesses operates its own enterprise solutions protected within a private network. The RFP notification should be received by a gateway service that can securely transfer and host the corresponding RFP document within the Proseware network to be picked up by one or more internal business processes. We assume that Proseware’s private network is extended into Azure using either a Site-to-site VPN or ExpressRoute.

Solution

Here we play the role of the Integration team at Proseware. We would like to future-proof the solution by creating a single internal integration point where one or more business applications running within the corporate network can pick the RFP for further processing. Simplifying it by storing the RFP in a Blob storage that can be accessed through the Proseware VNET.

We can setup a subscription on the Event Grid Topic. For event destinations, Event Grid supports Webhooks or Azure resources like Functions, Storage Queues, Event Hubs, Hybrid Connections or Service Bus. As this is a cross-enterprise integration scenario (Proseware and North Wind have separate Azure Subscriptions) we won’t be able to use the deep integrated Azure event destinations supported by Event Grid. That narrows down our choice to Webhooks.

The webhook subscription needs to point to a web service capable of doing the following.

1. Receive the webhook notification on a HTTP endpoint.
2. Parse the JSON based Event Grid schema to extract the RFP location on North Wind’s storage account.
3. Download the RFP from the North Wind storage account and place it on a Proseware storage account that can be accessed only via a VNET.

It is important to understand that Event Grid requires a notification confirmation within 30 seconds. In our case, the HTTP endpoint should be able to respond quickly. The design needs to ensure that RFP downloads in step 3 should not block the HTTP response as the downloads especially in case of batch events might take longer. Also note that this case does not require step 3 to make an authenticated call to fetch the RFP document from North Wind, we would like to use an approach that could be extended to support a custom authentication mechanism.

Azure Functions app with a HTTP trigger works well for us here. The ability to implement just the business logic in languages like C#, Python, PowerShell and JavaScript allows us to extend the functionality if needed. We are going to require access to the Proseware VNET, for which the Function app can be hosted on a Premium plan or a dedicated App Service Plan or App Service Environment which allow VNET integration. Here we opt for the App Service Plan as any spare compute can be shared by other functions apps or app services. If you prefer to benefit from serverless scaling, Premium Plan would be the way to go.^[7]

With Functions are now able to implement the service with minimal code (complete solution).We can leverage the Durable Functions extension to implement asynchronous activities in the functions app. The webhook can be pointed a HTTP trigger that initializes an orchestrator instance. The orchestrator can then use the fan-out/fan-in pattern to download each RFP in the list of events individually without blocking the webhook response.

[FunctionName("RFPNotificationOrchestrator")]
public static async Task<dynamic> RunOrchestrator(
    [OrchestrationTrigger] IDurableOrchestrationContext context)
{
    List<EventGridEvent> events = context.GetInput<List<EventGridEvent>>();
    var tasks = new List<Task<bool>>();

    foreach (EventGridEvent e in events) 
    {
        tasks.Add(context.CallActivityAsync<bool>("DownloadRFP", e));
    }
    
    await Task.WhenAll(tasks);
    return from t in tasks select new { TaskId = t.Id, Status = t.Result };
}

[FunctionName("DownloadRFP")]
public static async Task<bool> DownloadRFP([ActivityTrigger] EventGridEvent e,
[Blob("northwind-rfp-collection", Connection = "RFPStorageConnection")] CloudBlobContainer blobContainer,
ILogger log)
{
    log.LogInformation("Processing event {0}.",e.Id);
    // Check if event is for Webhook registration - https://docs.microsoft.com/en-us/azure/event-grid/webhook-event-delivery#validation-details
    if (e.EventType == "Microsoft.EventGrid.SubscriptionValidationEvent") {
        EventGridValidation validationObject = (e.Data as JObject).ToObject<EventGridValidation>();
        log.LogInformation("Validation URL: {0}", validationObject.validationUrl);
        return true;
    }

    RFPDescriptor descriptor = (e.Data as JObject).ToObject<RFPDescriptor>();
    log.LogInformation("Downloading RFP document ID {0} from North Wind", descriptor.id);
    var response = await _httpClient.GetAsync(descriptor.xml);
    var responseString = await response.Content.ReadAsStringAsync();

    // Store RFP to Blob Storage
    string blobName = String.Format("{0}.xml", descriptor.id);
    log.LogInformation("Uploading RFP document ID {0} to internal storage", descriptor.id);
    CloudBlockBlob blob = blobContainer.GetBlockBlobReference(blobName);
    await blob.UploadTextAsync(responseString);

    return true;
}

After implementing the integration point, Proseware is able to grant internal business processes access to the Blob storage to pick up fresh RFPs coming in from North Wind.

Proseware’s Integration Point to North Wind’s RFPs

Evolving processes

Businesses units like Sales, within Proseware would like to own and control their processes to reduce operational overheads. The solution needs to provide enough flexibility for the business process to evolve quickly. To begin with, the sales team would like to receive an email notification when a RFP is uploaded to the Proseware RFP Store (Blob storage). The email should include the specifics mentioned in the XML document in human readable format.

Solution

Considering that businesses would like to own and control the process here, it is ideal to pursue a low-code or no-code solution that businesses can easily maintain. The flexibility offered by Power Platform for such cases is incredible. But we are dealing with a scenario where RFPs are stored in a Blob storage controlled by Proseware with a private network.

Logic Apps on Azure could help, but they do not normally offer VNET integration that would allow us to access the Proseware Blob storage. However, a Logic App can be hosted in an Integrated Service Environment (ISE) - an offering from Azure specifically tailored for Logic Apps which offer the same level of resource isolation that App Service Environments (ASE) offer for App Services. This comes along with the ability to run specific actions with the ISE and leverage the provided VNET integration^[8].

Actions marked as CORE and ISE are executed in the ISE.

Further, we need to parse the XML file and extract the RFP specifications and place it in an email. This is where the Enterprise Integration Pack (EIP) comes in handy. It is important to note that EIP offers much more than just XML parsing. With EIP, an integration account can be used to store mapping instructions in form of Liquid templates, XSLT or HIDX. For XML to text, we will use the following Liquid template. You can learn more about Liquid here.

Hello team,
<br/><br/><br/>
North Wind has issued a notification requesting proposals against the following order.
<br/><br/>
-----------------------------------------------------------------------<br/>
Title: {{content.title}}<br/>
ID: {{content.id}}<br/>
Type: {{content.type}}<br/>
<br/><br/>
Order details:<br/>
{% for order in content.orders %}<br/>
{{order.name}} x {{order.quantity}}<br/>
{% endfor %}<br/><br/><br/>

{{content.note}}<br/><br/><br/>
-----------------------------------------------------------------------<br/>
Please follow up at the earliest.<br/><br/><br/><br/>



Kind regards,<br/>
Proseware Integration Team<br/>

If we use this template against the sample North Wind RFP XML, the end result looks like this.

Hello team,

North Wind has issued a notification requesting proposals against the following order.

-----------------------------------------------------------------------
Title: Intent to create the largest Lego Death Star
ID: IN934NZ11
Type: M582

Order details:

Darth Vader x 1

Storm troopers x 60

Emperor's Royal Guards x 20

Square blocks x 80000000

Blasters x 176

Vendor to take care of shipping and other logistics.

-----------------------------------------------------------------------
Please follow up at the earliest.

Kind regards,
Proseware Integration Team

We implement the following steps in the Logic App, starting with a blob file trigger followed by mapping the XML into text using the above template. The text is then sent over email to the sales team.

Logic App

With the full solution in order, we achieve the following:

1. North Wind’s ERP is able to publish events for vendors to consume.
2. Vendors like Proseware are able to create and manage their own subscriptions.
3. Proseware is able to pick RFP documents and make them accessible to internal processes.
4. The Sales team at Proseware is able to control and setup their own process like receiving emails with RFP details from North Wind.

Evolving Process

Disclaimer

The solution above is prepared with the intention to focus on the key serverless offerings from Azure. The reader is requested to revisit design decisions when building solutions to address their own requirements.

References

1. Marketing software company gives customers greater flexibility using serverless functions
2. E-discovery company uses serverless computing to speed internal development of its SaaS solution
3. The Urlist — An application study in Serverless and Azure
4. Choose between Azure messaging services - Event Grid, Event Hubs, and Service Bus
5. Azure Documentation - Event Grid Domains
6. Event Grid - Quotas and Limits
7. Azure Functions - Scale
8. Connect to Azure virtual networks from Azure Logic Apps by using an integration service environment (ISE)

Webinar Recording

PUG Live: Easing up Enterprise Integration with Azure Serverless

If you would like to stay up to date with new recording, subscribe to the Pune User Group youtube page

Proof of Identity is a primary problem for any network protocol and the World Wide Web (HTTP) is no different. Introduction of the state management RFC in 1997^[1] led to common use of Cookie headers to maintain a state of identity in HTTP sessions. These identities have been dominantly powered by logical authentication mechanisms. The world wide web has progressed far ahead in its adoption of identities that are authenticated by trusted delegates. These identities still aren’t universal over the world wide web. While these identities inter-operate, there’s isn’t a single central authority that controls these identities. To remain aligned with the philosophy of the open web, there shouldn’t be such a centralized entity. Here we discuss the possibility of extending cryptographic identities in the way they are being used in Distributed Ledger Technologies (popularly known as Blockchains), while keeping in mind the expected operational simplicity for an Internet user. Such identities would consistently keep the user in control and can be utilized by modern age applications to extend services with end-to-end encryption.

Considering the deprecation of the proposed keygen element as part of the HTML5 specification^[2], we discuss a ground up approach for cryptographic signatures for authentication on the web.

Cryptographic signatures for the Web

As opposed to conventional authentication that relies on username and password combinations, cryptographic authentication for the web can rely on messages signed by users with their secrets. The signing mechanism needs to fullfil the following two requirements.

1. Allow users to sign challenges posed by webpages to prove their identity.
2. Allow users to sign HTTP transactions that may occur through forms or XHR requests as a proof of authenticity.

Approach

1. Users will first have to generate their keys using standardized algorithms asymmetric cryptography.
2. Web pages can challenge the user to sign a message with their secret/private key.
3. Web services receive the signed challenge, and parse the signature for a public key, and verify the signature against the same public key.

Challenges can be used by web pages for registertion, login or even transactions. Public keys can be substituted as usernames for services that do not require user friendly naming conventions. Services like social platforms can map usernames with public keys for a friendly authentication experience.

Browser based interface

The standard HTML input field can be used as a base to build the signature component.

<input name="sign" type="signature" challenge="Hello Doctor Watson" />

On user action, the signature field generates a signature based on the form inputs filled by the user. The challenge attribute can be set by javascript or markup with information that needs to be signed.

Verification by services

Web services can receive HTTP form requests or XHR requests to parse the signature element in plain text for verification.

sendTo=omtalk
amount=3.14159
sign=-----BEGIN PGP SIGNATURE-----\nVersion: OpenPGP.js v2.5.10\nComment:https://openpgpjs.org\n\nwsFcBAEBCAAQBQJaSPPeCRCYcC33miZm0gAAiksQANEZS3XjrWx2Ah7v8aqs\nZXl7zURE8HPQn/9cv7M2/2bV82+1Kpnzr1\nnPhigQS4w2a7o/29jcMyrW0Bh\nI8N0R76sF3jVKiEY4YY5WJHu4HRJWi\nA9roUJRYPpvJgPAsCQgDb8HZF0v4x\nus/KJrT38xKQiCzqYbQCacqkWv/JuHCS4UA2D0RvCNy/xJeb750xJE2quNv9\nhkjffwnH49R1tzOm9zDkXtoQYxPBmrKCW8iKDbRIDSTs15kVIlBsmUEISKQN\nRH6B30rZZXulsY0dJjE6ovzsuGbR/Em8TefMu5pOzE4jvflqOu3hsVIkx1ZE\nd1/YuuJmbBy8pFWaj5efnqpSgCcCSSCAquAfXu2arh7LMD2yIO+mYrhA2gEv\n+8Yd6CrkgTRzKCE8D22qylbmlppJRPIfAutspvKvZrETcfJTpzONVUiFCjxt\nMnU7t9mn+frItQ4fBrHqfJyfNfAppIWqqnPZvr0iUsO4v2+JzlkNx/OtFjr6\nx6msgK87MzGgVzNPoLAWo054IDIWMO7YVrg2sB6NpsbtDhXAmXytloLfjcCx\nBWrl9Cpm2SWkRADg8IAB8FPnqbARdwJtlSQgG6A5v7FXwKEwA41uVkAkVyZs\nuoDEY3yri1FZF1JPlDnKT8CrKYQijkj2BXHcvDBo66hvxbalNQZvzOb02W1r\nHlBb\n=b++B\n-----END PGP SIGNATURE-----

The example uses PGP signature considering its widespread use by email messaging systems. The signature can be substituted by RSA, DSA or EC signatures as well.

Third-party authentication

Host services can procure signed contracts from users to obtain or delegate access to third-party services. Contracts shall be signed within scope of the host service. These contracts can then be forwarded to the third-party service for token generation.

Contracts are nothing but challenges that comply with a previously agreed upon structure. For example:

host: 0x8.in
delegate: twitter.com
token: -----BEGIN PGP SIGNATURE-----\nVersion: OpenPGP.js v2.5.10\nComment:https://openpgpjs.org\n\nwsFcBAEBCAAQB.....

Security measures

Transactions signed with cryptographic signatures can seem permanent in nature.

1. Signed transactions need to be secured within the scope of a single host at all times.
2. Signed transactions should not be replayable even within the scope of a single host.

Additional two parameters are prepended to all information being signed by the user.

host: 0x8.in
nonce: AzkiyxbYbnskb68B-1514731815897

• host: The root domain used by HTTP requests.
• nonce: A random challenge previously agreed upon by the browser and the host. The challenge can have the same expiry as the browser session.

Conclusion

The structure of identity and content ownership on the World Wide Web can be improved with the adoption of cryptographic proofs as a standard. The approach has proved fruitful for email signatures with PGP.

Moreover, cryptographic identities can provide the necessary foundation for end-to-end encryption (E2EE) required to ensure security and privacy of users of the world wide web. Further, the experimental Web Crypto API^[3] could be leveraged to standardize cryptographic algorithms paired with a browser based key store interface.

References

1. HTTP State Management Mechanism (rfc2109)
2. Keygen browser compatibility
3. Web Crypto API