Throttling on Dynamics 365 Finance & Operations (Supply Chain Management) Starting 10.0.19

Priority-Based Throttling is starting on version 10.0.19.  This has caused some concern for users who are heavily dependent on OData or Custom Web Service calls from third party software, i.e. ISVs, EDI, Integrations, web portals, custom web apps.  Everything here is up to date as of July of 2021.  I expect some changes to come over the next months and years which may invalidate some of what is discussed here.  I oversaw extensive testing, analyzed a month of LCS logs, and held some insightful conversations with Microsoft so there are a few gems in here for everyone.

Ξ Why Throttle Requests?  Throttling is a common way SaaS vendors prevent spikes in demand which can cause degradation of the user experience, or in the worst case server crash (denial of service).  Microsoft is responsible for providing an environment with a certain level of responsiveness, and therefore they must throttle lower-priority requests to ensure that the best possible user experience.

Ξ What is throttled?  In general, any third-party OData (F&O Data Entity) or Custom Service requests will be throttled.  Current exclusions/exemptions to throttling include:

• Interactive client experience in the web site
• Document Routing Agent (DRA)
• Warehouse Mobile (WHSMobile)
• Office Integration
• RetailAPI
• Data Import/Export Framework (DIXF) API
• Data Integrator
• Dual-write
• Finance and Operations Virtual Entities in the Power Platform
• Finance and Operations apps Connector (for Power Automate Flow, Power Apps, and Logic Apps)

Check Microsoft Docs for updates to the exclusion list in the future.

Ξ When does throttling kick in?  To accomplish the goal of reliable performance, F&O does not need to start throttling until the environment is under ‘heavy load.’  The F&O load balancer will look at the currently used CPU and memory to determine when to begin throttling for each priority tier.  The thresholds will vary from environment to environment, and this is an area that Microsoft wants to keep to their discretion.  For illustrative purposes only lets say the following thresholds are established.

• At 60% CPU or memory utilization, low priority requests will be asked to retry later
• At 70% CPU or memory utilization, low and medium priority requests will be asked to retry later
• At 80% CPU or memory utilization, all non-exempt requests  requests will be asked to retry later

I have found that currently F&O is consistently asking for the request to wait for 60 seconds before sending the request again.  You could retry after 10 seconds, but there isn’t a guarantee that it would work.  You could also try after five minutes, and the same is true.

Ξ Configuring Priorities Priorities should be configured in F&O to ensure that near-real time calls are given the priority over less critical requests.  If you leave something unconfigured, or forget to have a record for a particular user or client ID, it will default to high.

System administration > Setup > Throttling priority mapping

Suggestions

• Read the Docs, overview and FAQ, thoroughly
• Watch the FastTrack training on Getting Your F&O Environment Ready for Priority-based Throttling | April 14, 2021
• Each integration and ISV connection should have its own user account or AAD client registration to enable more granular priority configuration and better troubleshooting
• Have an environment similarly-sized to production where you can properly test throttling, i.e. Tier 4 or Tier 5.
• Validate that all Independent Software Vendors (ISVs) in use support throttling
• Validate all custom code supports throttling by handling the code 429 response and retrying the call later or using the DIXF/DMF API instead of OData calls
• Use the LCS queries to try and predict how many requests would be throttled with current loads.
• If you have a complex environment or additional questions, join the Dynamics Insiders then join the “Data Management, Data Entities, OData and Integrations” Yammer group to participate in the conversations and office hours Teams meeting going on (under NDA).

Tips and Notes

On versions prior to 10.0.19, I found the “Requests Throttled” report to be more accurate that the “All Throttling Events” query in LCS when looking for what requests could be subject to throttling when the environment is upgraded to 10.0.19+.

Caveat: When the system is utilized above the threshold, let’s say 70%, then it stops replying to requests based on the priority.  In that situation if the environment stays above 70% utilization for an hour, NONE of the low-priority requests will be processed during that time.  I see this as a possible area of improvement, as throttling could work more like airport queues, with dedicated processing power for all priorities so that low priority requests would be processed during heavy load, although at a reduced rate when compared to high-priority requests.  Now, before this sends up an alarm, this would be very atypical and rare.

If the LCS log analysis reveals that considerable requests may be throttled:

• Is there a pattern?  Could a heavy batch job (traditional resource planning, or BYOD, ….) be causing significant load and causing the throttling during a specific hour of each day?  If so, can interfaces be rescheduled or avoid that time period?
• Consider architecting the integrations to use the DMF/DIXF API or read data from the reporting solution which received Dynamics data via BYOD or data feed to Azure Data Lake.
• Could the environment be undersized?  Do not be surprised that Microsoft asks you to fill out the subscription estimate again.  It is Microsoft’s main tool for sizing environments and their baseline to compare and optimize a plethora of environments.  Take advantage of the telemetry data you can get from F&O and reporting tools to update the subscription estimate spreadsheet then submit a support ticket.

Good luck with your testing!  Please leave a comment or tweet me with feedback or your experiences.

Dag III

The post Throttling on Dynamics 365 Finance & Operations (Supply Chain Management) Starting 10.0.19 appeared first on Dag Calafell, III.

Original Post https://calafell.me/throttling-on-dynamics-365-finance-operations-supply-chain-management-starting-10-0-19/