Fix eventual consistency issues in App Registration, Group, and Service Principal resources

[ms-henglu]

fixes #1802

Community Note

• Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
• Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for PR followers and do not help prioritize for review

Description

Changes to existing Resource / Data Source

• I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
• I have written new tests for my resource or datasource changes & updated any relevant documentation.
• I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
• (For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

Testing

• My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

• azuread_resource - support for the thing1 property [GH-00000]

This is a (please select all that apply):

• Bug Fix
• New Feature (ie adding a service, resource, or data source)
• Enhancement
• Breaking Change

Related Issue(s)

Fixes #0000

Rollback Plan

If a change needs to be reverted, we will publish an updated version of the provider.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

[khenderson42]

Apologies if this isn't the right place for this feedback.

We forked this branch and pushed it to our private registry for testing as the issue it resolves has become a major blocker for us.

The code changes resolved the problems with app registrations, service principals and groups (thank you for that), but the same problem persists with azuread_group_member, azuread_directory_role_assignment and azuread_app_role_assignment.

I'm not a Golang dev, so I stole then tweaked the consistency changes from this PR and applied them to azuread_group_member and azuread_app_role_assignment and it seems to have resolved the issue for us. I've not tried on azuread_directory_role_assignment yet.

[jaredfholgate]

@ms-henglu you are probably already aware, but I think this needs applying to every resource.

[vdidenko-dvps]

When we can expect this fix to be released? Thank you!

@jaredfholgate

[jaredfholgate]

When we can expect this fix to be released? Thank you!

@jaredfholgate

I'm afraid I can't comment on the timescales. @ms-henglu is aware of the urgency and trying hard to get something out ASAP.

[travisgan]

@ms-henglu you are probably already aware, but I think this needs applying to every resource.
comment from @jaredfholgate

@ms-henglu, thanks for the new commit but seems that is only to address azuread_group_member, azuread_directory_role_assignment and azuread_app_role_assignment.

The fix appears need to be applied to every resource, eg. azuread_application_permission_scope, azuread_application_app_role and etc. Those are having similar issue as well.

[jaredfholgate]

@ms-henglu you are probably already aware, but I think this needs applying to every resource.
comment from @jaredfholgate

@ms-henglu, thanks for the new commit but seems that is only to address azuread_group_member, azuread_directory_role_assignment and azuread_app_role_assignment.

The fix appears need to be applied to every resource, eg. azuread_application_permission_scope, azuread_application_app_role and etc. Those are having similar issue as well.

It's not finished yet, a review will be requested when it is ready to be merged.

[KrisBoeckx]

Any update on this issue ? I'm facing this issue and can't provision anything. If there is a work arround, please tell us. If we have to wait, please tell us.

[airahyusuff]

Any update on this issue ? I'm facing this issue and can't provision anything. If there is a work arround, please tell us. If we have to wait, please tell us.

@KrisBoeckx maybe try pinning the provider version to v3.6.0, this helped me move along with my work until the issue is resolved.

[KrisBoeckx]

Any update on this issue ? I'm facing this issue and can't provision anything. If there is a work arround, please tell us. If we have to wait, please tell us.

@KrisBoeckx maybe try pinning the provider version to v3.6.0, this helped me move along with my work until the issue is resolved.

Reverting back to version 3.6 is what I first did. But it did not resolve the issue.
The problem is that there was a change on the Microsoft GRAPH API side. Reverting back to an older version like 3.6 will not solve that issue...

[MvRoo]

Any update on this issue ? I'm facing this issue and can't provision anything. If there is a work arround, please tell us. If we have to wait, please tell us.

The problem is one of eventual consistency, meaning Entra / MS Graph will become eventually consistent at some point (often pretty soon), but due to the error, Terraform has marked the resource as tainted and will proceed to destroy and recreate it, creating a never-ending loop. If you are not automating this at scale, you could apply a manual workaround by updating the tf state file and then running the tf apply again.

[SkyDrive26]

Any update on this issue ? I'm facing this issue and can't provision anything. If there is a work arround, please tell us. If we have to wait, please tell us.

@KrisBoeckx maybe try pinning the provider version to v3.6.0, this helped me move along with my work until the issue is resolved.

Reverting back to version 3.6 is what I first did. But it did not resolve the issue. The problem is that there was a change on the Microsoft GRAPH API side. Reverting back to an older version like 3.6 will not solve that issue...

Our workaround for now has been to fork the PR in our pipelines and then build it on our agents for a local dev override in Terraform. This allowed us to work with the updates in this PR prior to the release. Now this won't be ideal to run in a sensitive production environment, but it works for us since we only face this issues in a setup which is in MVP phase anyway.

[cvs79]

Thanks for putting this together. Have forked the PR and built a local version of the provider, which has unblocked us significantly.

Appreciate the work done. Will continue testing and follow up with any feedback.

[DerGary]

would be nice to have this released as soon as possible

[MaxPowers1337]

Is anyone terraform enterprise customers perhaps? He could request this issue to your corresponding technical account manager or something like this.

[markti]

this would be a lovely christmas present... 🎁 😅

[jaredfholgate]

The Entra ID API change has been reverted for Terraform, which should have solved this in the short term. It will be fully rolled out to all regions by end of today. This PR or a similar solution will still be implemented as a permanent solution.

[overfl0wd]

@jaredfholgate Deployments are now working for groups, app registrations, app assignments etc, but the inconsistency issue is still present for conditional access policies - azuread_conditional_access_policy. It would be greatly appreciated if those APIs could be reverted as well. Thank you!

[jaredfholgate]

@jaredfholgate Deployments are now working for groups, app registrations, app assignments etc, but the inconsistency issue is still present for conditional access policies - azuread_conditional_access_policy. It would be greatly appreciated if those APIs could be reverted as well. Thank you!

Hi. This is being looked into, it is possible this uses a different API endpoint. Attempting a reproduction today as need correlation IDs, etc. Thanks

[jaredfholgate]

@jaredfholgate Deployments are now working for groups, app registrations, app assignments etc, but the inconsistency issue is still present for conditional access policies - azuread_conditional_access_policy. It would be greatly appreciated if those APIs could be reverted as well. Thank you!

I've been unable to reproduce this issue after multiple attempts. Please can you share an example policy that exhibits the issue? If you can also share any log details such as a correlation ID and timestamp that would be helpful. Thanks

[tvandinther]

@jaredfholgate I'm experiencing a potential session affinity issue with the azuread_app_role_assignment resource. The assignments are being created and appear in the audit logs on the Azure portal but the provider is seemingly unable to import it back into state with the subsequent GET request. This issue is happening on about 5-10% of total cases. In this last case it happened in 2 out of 4 role assignments sent off in one state change (for one app).

Note: I am using this Terraform provider through Crossplane's provider-upjet-azuread.

Correlation ID 1: f1577cd0-a6c8-4e31-9518-40cf11b5bba9
Correlation ID 2: 44c2c699-c7e1-4873-97d3-5775a7f08ae4

[dchauran-skillable]

Is the API rollback still in place? I can consistently reproduce this issue with simple app/SP creation currently, and would love to see this fix in place.

[danlumb01]

I'm seeing this issue when attempting to create an app registration with a service principal (azuread_application and azuread_service_principal are the specific resouces I'm using).

[Kerwood]

I haven't had the issue since @jaredfholgate posted that they fixed it.

[BOLT04]

I have had these same eventual consistency issues, but in the Graph API directly. It can take several minutes and I don't see any documentation stating how long it could take. Replication latency has been very high for me in the past month. From what I can tell from Azure Entra docs it's easier to replicate this problem with application-only requests. Because these don't have session consistency:

Writes are immediately replicated to the secondary replica to which the logical session's reads were issued.
For application-only requests, Microsoft Entra ID does not provide session consistency. Session consistency is only supported for delegated requests (application+user token flow).

[dixie-flatline999]

I am suffering the inconsistency issue for azuread_group_member up to this day

[joshwright10]

Are there any plans to continue working on this fix?
We just came across this today with the azuread_app_role_assignment resource.
azuread v3.9.0

[vdidenko-dvps]

Hi,
@jaredfholgate

I've ran into same issue today 3 times for same change promoted in 3 environments with azurerm_user_assigned_identity

Seems someone reverted the fix?

I haven't provisioned new azurerm_user_assigned_identity for a while, so can't locate when this has started

[jaredfholgate]

Hi, @jaredfholgate

I've ran into same issue today 3 times for same change promoted in 3 environments with azurerm_user_assigned_identity

Seems someone reverted the fix?

I haven't provisioned new azurerm_user_assigned_identity for a while, so can't locate when this has started

Hi @vdidenko-dvps

The resource you mention is part of the azurerm provider and interacts with the ARM endpoint, not the Graph endpoint. If you are having an issue with this resource, you should report it over here along with some details like the error you are seeing: https://github.com/hashicorp/terraform-provider-azurerm/issues

[tvandinther]

This problem has been plaguing production automations for almost a year. What is the current plan to resolve this? Is there something wrong with the proposal in this PR? Is this being worked on somewhere else?

Some information would be gladly appreciated. In the meantime I have seen the original issues return over the last few months as the "temporary rollbacks" have seemingly been removed.

[Kerwood]

I haven't experienced any issues since @jaredfholgate posted the fix last December, despite deploying a large number of App Registrations, app roles, role assignments, groups, and related resources.

I think this issue is being blamed for a number of unrelated problems.

For example:

I am suffering the inconsistency issue for azuread_group_member up to this day

This could very well be related to a different issue using the beta endpoint, that also causes inconsistencies: #1899

Or take @vdidenko-dvps's issue, which isn't even using the Microsoft Graph endpoint.

[dixie-flatline999]

This could very well be related to a different issue using the beta endpoint, that also causes inconsistencies: #1899

That is quite an assumption. I have exactly the inconsistency issues that this PR solves and see them every week at minimum, as our pipelines manage hundreds of group memberships, using version 3.8.0.

For now I forked the repo and merged this PR to see if the issue disappears.