stuck-tombstoned-row

name: stuck-tombstoned-row description: > Investigates reverse migration stuck at WaitUntilTombstonedLogicalDatabaseRowIsRecycled. Follows TSG SOC074.8. The tombstoned logical database row cannot be deleted due to a FK constraint violation from a stale scheduled UpdateSlo request.

SOC074.8 — Reverse Migration Stuck at WaitUntilTombstonedLogicalDatabaseRowIsRecycled

Owner: hsreversemig@microsoft.com

Keywords: VldbReverseMigrationRequestStateMachine, WaitUntilTombstonedLogicalDatabaseRowIsRecycled

Symptom

• Reverse migration stuck at state WaitUntilTombstonedLogicalDatabaseRowIsRecycled.
• The tombstoned logical database row cannot be recycled (deleted) from CMS.

Prerequisites

• Parent triage skill: Socrates\hs-reverse-migration\triage
• XTS view: sterling\sterling cms browser v2.xts

Investigation

Case 1: FK_scheduled_update_slo_requests_logical_databases

This case applies if the following exception is hit:

The DELETE statement conflicted with the REFERENCE constraint 
"FK_scheduled_update_slo_requests_logical_databases". The conflict occurred in database 
"ClusterMetadataStore", table "dbo.tbl_scheduled_update_slo_requests".
The statement has been terminated.

Explanation:

• In the WaitUntilTombstonedLogicalDatabaseRowIsRecycled state, the workflow tries to delete the row from dbo.tbl_logical_databases with the key (logical_server_name, logical_database_id).
• The FK violation means there is a row in dbo.tbl_scheduled_update_slo_requests referencing the same key.

Conditions required for this case (all must be met):

1. UpdateSLO from higher DTU-based premium SLO to lower DTU-based premium SLO
2. Forward migration from Sterling to Hyperscale
3. Reverse migration from HS to GP, within 3 days of Step 1

Mitigation

Case 1: FK_scheduled_update_slo_requests_logical_databases

Run CAS command to remove the blocking row in dbo.tbl_scheduled_update_slo_requests:

Stop-ScheduledUpdateSlo -DatabaseId <logical_database_id> -ServerName <logical_server_name> -IncidentId <Icm>

Example:

Stop-ScheduledUpdateSlo -DatabaseId DFCD6374-A579-4CF0-880D-771AD4334FB5 -ServerName mel0204adssql01 -IncidentId 689392885

After running this command, the reverse migration workflow should proceed and complete.

Repair Items

Reference ICMs

• ICM 689392885