Custom APIs and schemaversion 2.0

ajkauffmannBusiness Central6 months ago20 Views

With Business Central 2024 wave 1 (v24) Microsoft changed the behavior of custom APIs that affected almost every API integration. And many didn’t see that coming, including myself. Well, I knew it was coming, but I wasn’t really prepared for the impact. I’ve seen quite some messages on forums and social media from others struggling with it and also received a few direct messages. Although it’s already a few months ago, I figured it makes sense to document what happened and how to deal with it.

What happened?

Starting with version v24, custom APIs switched from schema version 1.0 to 2.0. Before we dive into the details and why it was a breaking change, let’s first look at the history of the illustrious schema version parameter.

In Business Central April 2020 wave 2 (v17) API v2.0 was introduced, together with schema version 2.0. For API v2.0, the schema version is always set to 2.0 ‘to get the latest features in the Business Central OData stack‘. For custom APIs, the default schema version was always 1.0. The reason for this discrepancy between standard APIs and custom APIs was never explained, the only comment was that for custom APIs it was just a matter of adding ?$schemaversion=2.0 to the URL.

In the overview of deprecated features in the platform for version 2023 release wave 2 (v23), there was a warning that starting with v24, the default value of $schemaversion will be 2.0 also for custom APIs. (source). And in v24, this change became a reality (source).

In hindsight, this wasn’t the most brilliant move from Microsoft. The change to schema version 2.0 was definitely a breaking change (as I will show in this article) and we got the warning just half a year in advance instead of the usual full year. And as far as I remember, there wasn’t a lot of communication around this. And why did we have a different behavior between standard and custom APIs, to begin with?

Microsoft could (and does) say like: ‘Hey, you can easily revert to schema version 1.0 by adding $schemaversion=1.0 to the URL’. But for many integrations that isn’t so easy, because they may have been created by 3rd parties or simply don’t offer the option to add this query parameter to the URL. What’s more, even Microsoft’s own Business Central connector for Power Automate doesn’t support an option for setting the schema version. As a result, many Power Automate flows using custom APIs failed after the upgrade to v24.

Schema version 2.0

Anyway, what’s the deal with schema version 2.0?

The difference between schema version 1.0 and 2.0 is the way how Enums are handled. I wrote about this in December 2020 (What’s new in Business Central API v2.0). But let me repeat it here and dig a little deeper.

In schema version 1.0, Enums are exposed as a string. This is how the field blocked in the customers API is defined in schema version 1.0:

<Property Name="blocked" Type="Edm.String" />

This is the response of the customers API in schema version 1.0:

{
    "value": [
        {
            "number": "10000",
            "displayName": "Adatum Corporation",
            "blocked": " "
        },
        {
            "number": "20000",
            "displayName": "Trey Research",
            "blocked": "Ship"
        },
        {
            "number": "30000",
            "displayName": "School of Fine Art",
            "blocked": "Invoice"
        },
        {
            "number": "40000",
            "displayName": "Alpine Ski House",
            "blocked": "All"
        },
        {
            "number": "50000",
            "displayName": "Relecloud",
            "blocked": " "
        }
    ]
}

In schema version 2.0 the field blocked is a strong type instead of a string:

<Property Name="blocked" Type="Microsoft.NAV.customerBlocked" />

The type Microsoft.NAV.customerBlocked refers to this definition in the metadata:

<EnumType Name="customerBlocked">
    <Member Name="_x0020_" Value="0" />
    <Member Name="Ship" Value="1" />
    <Member Name="Invoice" Value="2" />
    <Member Name="All" Value="3" />
</EnumType>

There you have the difference, in schema version 2.0 Enum fields are exposed as strong type instead of a simple string. A strong type means it has stricter rules, usually checked during compilation. In the context of an API, the Enum type exposes the possible values in the metadata to enforce strict rules on which values are accepted.

Here is the result of the customers API in schema version 2.0:

{
    "value": [
        {
            "number": "10000",
            "displayName": "Adatum Corporation",
            "blocked": "_x0020_"
        },
        {
            "number": "20000",
            "displayName": "Trey Research",
            "blocked": "Ship"
        },
        {
            "number": "30000",
            "displayName": "School of Fine Art",
            "blocked": "Invoice"
        },
        {
            "number": "40000",
            "displayName": "Alpine Ski House",
            "blocked": "All"
        },
        {
            "number": "50000",
            "displayName": "Relecloud",
            "blocked": "_x0020_"
        }
    ]
}

Now you would think that the weird _x0020_ value is the only difference between schema version 1.0 and 2.0. But that’s not true! There is one more difference that goes completely unnoticed in the examples shown so far.

In schema version 1.0, the enum fields return the en-US caption of the enum value. While in schema version 2.0 the enum fields return the name of the enum value. I believe this is a little-known fact about APIs and enums.

Consider the following enum, to be used on Customer records.

enum 50102 CustomerLevel
{
    value(0; BASIC) { Caption = 'Basic Level'; }
    value(1; BRONZE) { Caption = 'Bronze Level'; }
    value(2; SILVER) { Caption = 'Silver Level'; }
    value(3; GOLD) { Caption = 'Gold Level'; }
    value(4; PLATINUM) { Caption = 'Platinum Level'; }
}

As you can see, the enum names and captions are different. That is not according to the common coding conventions, but it perfectly demonstrates the difference between schema version 1.0 and 2.0.

This is the result with schema version 1.0 (the default for custom APIs up to v23):

{
    "value": [
        {
            "id": "971d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "10000",
            "name": "Adatum Corporation",
            "customerLevel": "Gold Level"
        },
        {
            "id": "9b1d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "20000",
            "name": "Trey Research",
            "customerLevel": "Basic Level"
        },
        {
            "id": "9f1d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "30000",
            "name": "School of Fine Art",
            "customerLevel": "Platinum Level"
        },
        {
            "id": "a31d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "40000",
            "name": "Alpine Ski House",
            "customerLevel": "Bronze Level"
        },
        {
            "id": "a71d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "50000",
            "name": "Relecloud",
            "customerLevel": "Silver Level"
        }
    ]
}

And this is the result with schema version 2.0 (the default from v24):

{
    "value": [
        {
            "id": "971d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "10000",
            "name": "Adatum Corporation",
            "customerLevel": "GOLD"
        },
        {
            "id": "9b1d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "20000",
            "name": "Trey Research",
            "customerLevel": "BASIC"
        },
        {
            "id": "9f1d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "30000",
            "name": "School of Fine Art",
            "customerLevel": "PLATINUM"
        },
        {
            "id": "a31d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "40000",
            "name": "Alpine Ski House",
            "customerLevel": "BRONZE"
        },
        {
            "id": "a71d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "50000",
            "name": "Relecloud",
            "customerLevel": "SILVER"
        }
    ]
}

Do I need to explain more? This IS a breaking change, right? At first glance, the only difference looks like spaces are being encoded like _x0020_ and that’s it. But this is not just about formatting, the APIs are now returning the enum name instead of the caption. The reason why it only looks like a formatting issue is because of the common coding convention to use spaces and other punctuation marks in names so they match the English caption.

On a side note, what if AL didn’t allow using spaces or other punctuation marks in identifiers? Imagine, no spaces, dots, or slashes in object names and field names. I never understood why a field name should be the same as its English caption. I wouldn’t shed a tear if the coding convention for AL would be changed so that we can only use alphanumeric characters and numbers in identifiers. So, there I said it…

Summary

Ok, let’s summarize what we have seen so far:

Schema version 1.0 returns enum captions
Schema version 2.0 returns enum names
Punctuation marks in enum names are encoded with _x

Unicode formatting

But what is that _x0020_ doing there? It is a format for escaping Unicode characters. In schema version 2.0 punctuation marks are formatted as escaped Unicode.

What is unclear to me is why these values must be translated into this escaped Unicode format. JSON data always uses the Unicode character set anyway, and the punctuation marks we are talking about are just ASCII characters. Why would it be fine in schema version 1.0 to have spaces and slashes and whatnot, but suddenly in schema version 2.0 that would be a problem? I find it hard to believe. My best guess would be that by formatting it in this way, the values exposed in the metadata are compatible across any language, platform, and operating system. If another system would retrieve the possible enum values from the metadata, maybe they would expect only alphanumeric characters, so spaces and other punctuation marks could be a problem. Did I already say I’m a fan of names without spaces?

On top of that, the format _x is not a common Unicode escaped format for JSON as far as I know. Instead, the format \u (e.g. \u0020) is normally used for formatting Unicode characters in JSON data. I have checked with some JSON tools, and they have no idea how to handle the _x format, but they do handle the \u format correctly. The _x format however is commonly used in XML for Unicode formatting.

So, let me reveal how the platform translates the enum names: it is using XmlConvert.EncodeName(). This is a function that translates invalid XML characters into escaped numeric encodings. More info about this function can be found here.

And let me be honest, I’m really confused now. The enum names in JSON are encoded as XML. Why would we need that? The EncodeName function is used to encode XML element names. But we are talking about enum names that are never being exposed as XML element names!! They are always exposed as values, even in the API metadata. They are never being used as real XML element names. So why encode them as valid XML element names? And if there is a reason, why not use proper JSON encoding? I’m baffled and have no explanation…

Microsoft, if you read this, please consider schema version 3.0 and remove the XML encoding or (if it really must be encoded) switch to the \u formatting. Pretty please…

Anyway, for now, this is the default behavior for both standard and custom APIs. So now you know it.

POST and PATCH requests

All of the above was about the JSON returned by the APIs. But what about POST and PATCH requests? Is there any change between the schema versions, do we also need to provide Unicode-encoded values?

The short and simple answer is: No, that is not necessary. POST and PATCH requests accept enum values without encoding and also enum captions. That was the case in schema version 1.0 and it still is in version 2.0. Below is a picture of a PATCH request to a custom API using an enum caption, while the response returns the enum name.

But if you want to post values with the Unicode encoding, then you must use $schemaversion=2.0. They are not supported with version 1.0. But now that version 2.0 is the default for all APIs, there is no need to add the parameter $schemaversion to the URL.

Handling _x characters

The next question is how to deal with these _x characters. What are your options? Let me give you some ideas on how to get around that.

Option 1: Enforce schema version 1.0

If possible, the easiest option would be to enforce schema version 1.0 by adding ?$schemaversion=1.0 to the URL. Then you will get the enum captions instead of the values, without encoded characters.

Option 2: Use XmlConvert

If your integration is written in C#, you may use XmlConvert.DecodeName(value) to get the original value without the _x characters.

For posting data, you may use XmlConvert.EncodeName(value) to match the encoded enum name.

Option 3: Decode / Encode in Power Automate

If you are using Power Automate, you have probably found out that the _x values are treated as text. If you want to get the original value, you need to convert it. Power Automate supports that with the function DecodeXmlName(value). Of course, there is also a function EncodeXmlName(value) to convert an enum name to the encoded format.

Option 4: Use the provided translations

This option is suitable for integrations that are using a UI. If enum names should be displayed in the UI, then you could of course convert the encoded values. But you can also use the endpoint /entityDefinitions which returns the translated captions of all exposed fields, including enums. The example above for enum CustomerLevel looks like:

{
    "entityName": "customerLevel",
    "entityCaptions": [],
    "entitySetCaptions": [],
    "properties": [],
    "actions": [],
    "enumMembers": [
        {
            "name": "BASIC",
            "value": 0,
            "captions": [
                {
                    "languageCode": 1033,
                    "caption": "Basic Level"
                }
            ]
        },
        {
            "name": "BRONZE",
            "value": 1,
            "captions": [
                {
                    "languageCode": 1033,
                    "caption": "Bronze Level"
                }
            ]
        },
        {
            "name": "SILVER",
            "value": 2,
            "captions": [
                {
                    "languageCode": 1033,
                    "caption": "Silver Level"
                }
            ]
        },
        {
            "name": "GOLD",
            "value": 3,
            "captions": [
                {
                    "languageCode": 1033,
                    "caption": "Gold Level"
                }
            ]
        },
        {
            "name": "PLATINUM",
            "value": 4,
            "captions": [
                {
                    "languageCode": 1033,
                    "caption": "Platinum Level"
                }
            ]
        }
    ]
}

Option 5: Let the API return the caption of enums like it was in schema version 1.0

This is a simple one for read-only APIs. Just put Format() around the field and you will get the caption instead of the enum name. But be careful, with this option, you can’t filter the field anymore! (Thanks to anonymous comment). If you need to filter on the enum value, then consider adding the enum field twice: one with the raw value for filtering and another one with the format.

field(customerLevel; Format(Rec.CustomerLevel)) { }

And now the result in schema version 2.0 is the same as with version 1.0:

{
    "value": [
        {
            "id": "971d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "10000",
            "name": "Adatum Corporation",
            "customerLevel": "Gold Level"
        },
        {
            "id": "9b1d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "20000",
            "name": "Trey Research",
            "customerLevel": "Basic Level"
        },
        {
            "id": "9f1d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "30000",
            "name": "School of Fine Art",
            "customerLevel": "Platinum Level"
        },
        {
            "id": "a31d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "40000",
            "name": "Alpine Ski House",
            "customerLevel": "Bronze Level"
        },
        {
            "id": "a71d885c-2f54-ef11-bfe7-6045bdacaf75",
            "number": "50000",
            "name": "Relecloud",
            "customerLevel": "Silver Level"
        }
    ]
}

But be careful, by using Format() the field becomes read-only. You can’t set the field value with a POST or PATCH request. If you also need to set the field value, then the next option is for you.

Option 6: Return caption and support both read and write requests

This is a variant of option 5 that supports GET, POST, and PATCH requests. It returns the enum caption as in schema version 1.0 and you can set the enum field with either the enum name or caption. Again, filtering is not possible on this field!

page 50101 CustomerAPI
{
    PageType = API;
    APIPublisher="ajk";
    APIGroup = 'demo';
    APIVersion = 'v1.0';
    EntitySetName="customers";
    EntityName="customer";
    DelayedInsert = true;
    SourceTable = Customer;
    ODataKeyFields = SystemId;

    layout
    {
        area(Content)
        {
            field(id; Rec.SystemId) { }
            field(number; Rec."No.") { }
            field(name; Rec.Name) { }
            field(customerLevel; customerLevelTxt)
            {
                trigger OnValidate()
                begin
                    Evaluate(Rec.CustomerLevel, customerLevelTxt);
                end;
            }
        }
    }

    var
        customerLevelTxt: Text;

    trigger OnAfterGetRecord()
    begin
        customerLevelTxt := Format(Rec.CustomerLevel);
    end;
}

Option 7: Return enum name without encoding and support both read and write requests

This is a variant of option 6. Instead of returning the enum caption, it returns the enum name, but without the encoding. And it supports write requests as well. Best of both worlds, right? But again, filtering is not possible on this field!

page 50101 CustomerAPI
{
    PageType = API;
    APIPublisher="ajk";
    APIGroup = 'demo';
    APIVersion = 'v1.0';
    EntitySetName="customers";
    EntityName="customer";
    DelayedInsert = true;
    SourceTable = Customer;
    ODataKeyFields = SystemId;

    layout
    {
        area(Content)
        {
            field(id; Rec.SystemId) { }
            field(number; Rec."No.") { }
            field(name; Rec.Name) { }
            field(customerLevel; customerLevelTxt)
            {
                trigger OnValidate()
                begin
                    Evaluate(Rec.CustomerLevel, customerLevelTxt);
                end;
            }
        }
    }

    var
        customerLevelTxt: Text;

    trigger OnAfterGetRecord()
    begin
        customerLevelTxt := Rec.CustomerLevel.Names.Get(Rec.CustomerLevel.Ordinals.IndexOf(Rec.CustomerLevel.AsInteger()));
    end;
}