We started encountering this issue on our on-prem (Hybrid) exchange server all of a sudden which was preventing emails from being scanned by the anti malware module, and hence were being held in the submission queue and not being delivered.
You may see errors in the event log such as;
The FIP-FS Scan Process failed initialization. Error: 0x80010105. Error Details: The server threw an exception.
The FIP-FS Scan Process failed initialization. Error: 0x800706BE. Error Details: The remote procedure call failed.
Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0
Problem signature:
P1: scanningprocess.exe
The usual server and service restarts did not fix the problem.
The issue appears to be related to updates not being installed correctly for the exchange anti malware scanning module.
You can check/confirm this by running these commands from Powershell;
Add-PsSnapin Microsoft.Forefront.Filtering.Management.Powershell
Get-EngineUpdateInformation
We got a response like below - note the "UpdateAttemptFailed" status
You can also check the update engine settings by running the command
get-engineupdatecommonsettings
Note the PrimaryUpdatePath - when we attempted to access this URL we were getting a 404 error - this is potentially what the problem is?
Update 4/1/22
I found the following article from the Exchange Team Blog that references this issue and provides a script to fix it
Before implementing the provided fix, I can ran the default script to Enable-AntimalwareScanning (located in C:\Program Files\Microsoft\Exchange Server\V15\Scripts) - mail delivery operated for a brief period of time before mails began queuing again.
I ran the script provided in the blogpost, and even after a full server restart, the problem still persists so I have disabled the antimalware scanning again
Fix/Solution/Workaround
Disable the anti-malware scanning option. This is easily done by running the included powershell script in your Exchange installation directory
C:\Program Files\Microsoft\Exchange Server\V15\Scripts\Disable-AntimalwareScanning.ps1
Run the script then restart the Microsoft Exchange Transport Service
Once we did this the submission queue immediately started clearing.
You can reverse the change easily by running the Enable-AntimalwareScanning.ps1 script in the same folder later on.
We've also found this recent MS article outlining steps to manually update the scan engines used by Exchange Server, but we found it still failed to update the signatures after running it
https://docs.microsoft.com/en-us/exchange/troubleshoot/setup/manually-update-scan-engines
