Problem
The fact is that SharePoint 2010 servers think they need to be able to reach out to http://crl.microsoft.com. They need get a current list of certificates for revoked (expired, key compromised, CA compromised, etc.) certificates.
But that's not an option for most of the SharePoint servers I work with. It's common for firewalls that prevent such outbound http calls and therefore prevent CRL checking. The inability to check the CRL (certificate revocation list) can create a myriad of strange performance problems and timeouts for w3wp.exe's, owstimer.exe, asp.net, and .net applications. Sometimes those problems are pretty easy to figure out with a bit of research (like a 15 second delay when browsing to a SharePoint page after an IISRESET) and sometimes the problems that show up are more mysterious.
There are dozens of blogs on this subject and a variety of approaches people take. So why write on this again? There are at least eight somewhat bewildering options that Admins may be forced to choose from. Sometimes one of those options works well for one farm and doesn't work well for another farm with a similar problem. Sometimes the impulse to get the problem fixed is stronger than the impulse to abide by best security best practices. Some of the advice is dated and deserves improvement. Also it's not clear what applies to SP2007, to SP2010, and/or to SP2013. I'm going to attempt to consolidate and evaluate all the options.
Detection
If a SharePoint server doesn't have access to the CRL, I expect that you've got some kind of CRL checking problem to one degree or another.
One good way to determine whether or not you need to consider one or more of these options for your servers is to set up CAPI2 logging. It's not enabled by default but it's easy to do. Steps can be found at http://support.microsoft.com/kb/2625048. (Also see http://www.microsoft.com/en-us/download/details.aspx?id=18844)
Another way that this problem is often detected is for a fiddler/netmon/wireshark trace is performed while reproducing the problem and calls to crl.microsoft.com are noticed. Sysinternals Utilities such as Process Monitor and Process Explorer can also show this traffic. It might be good to restart IIS and restart the SharePoint timer service while doing this.
You could also task your network administrator with checking the firewall/proxy logs (in a 72 hour period?) to see which servers are making calls to crl.microsoft.com.
Option 1 - Adjust a FIREWALL RULE to permit traffic from the servers to crl.microsoft.com
When a SharePoint Administrator asks a Network Administrator to open a hole in a firewall, the Network Admin usually laughs a knowing-laugh and says, "Not unless you can give me a really good reason!" The answer of, "So SharePoint will perform better," isn't usually good enough. But I'd highly recommend revisiting this option.
Try asking the network admin / security engineer to look at the logs produced by the firewall/proxy to get a sense of the frequency of calls to crl.microsoft.com. This information also would show which servers are making these calls. And from there perhaps a firewall rule can be adjusted to be adequate-but-still-strict.
This option is the most effective solution and it ironically may also be the most secure options. It avoids the problem of disabling the CRL checking—something that does tend to be a good idea from a security standpoint. Most of the other options in this list disable CRL checking in one way or another and that's not really a best security practice. I'm not the only one saying this. Consider these quotes:
- "Note that disabling CRL check is not recommended in a production environment, unless you are troubleshooting an issue and wants to isolate if the problem is related to CRL validation. Make sure to turn it on again after performing the validation. . . While it is important to keep secure access to Internet from the enterprise standpoint, it is also important to make sure that applications that require Internet access to validate CRL are capable of doing it. CRL checking is a secure mechanism that helps validate the validity of a certificate. . . make sure to add crl.microsoft.com to the list (on your firewall or proxy) and you should avoid issues of this nature." (Reference)
- ". . . disabling this option is not recommended for security reasons" (Ref.)
- "Unblock access to the above URL so CRLs can be retrieved when needed. This is the *PREFERRED* approach." (Ref.)
- "PREFERRED RESOLUTION: Allow the server and the services accounts to access the crl.microsoft.com domain." (Ref.)
Option 2 - Add crl.microsoft.com to the HOST FILE and point it to the local address
It is very common for a SharePoint Admin to add crl.microsoft.com to the host files of their SharePoint servers and point it to 127.0.0.1. This is certainly easy to do and is usually enough to keep the biggest, most obvious sharepoint performance problems problems at bay. However, the CRL is not at 127.0.0.1. So right there I think this is not the best possible solution. The processes on the server that want to check the CRL still cannot check the CRL. Maybe you avoid having to wait for a timeout this way but maybe there are better options for both performance, security, and thoroughness. Several people report that this isn't always good enough to stop all crl checking related problems.
Option 3 - Set generatePublisherEvidence to FALSE in in a .config file
This option is easy to implement, is highly recommended by many reputable sources, tends to give really good results (solves problems that the host file entry doesn't always solve), and has the added advantage of being able to be applied with impressive granularity. For example, it can be applied in a web.config file for a specific web application, a root web.config file so that many/all web applications are affected, or in an owstimer.exe.config file to only helps the owstimer.exe process.
SharePoint uses .NET and ASP.net. .NET applications (all managed code) attempts to download the CRL for any signed assembly it loads on startup. This is normal. Setting generatePublisherEvidence to false prevents that.
Should it be done? I note how MSDN seems to recommend it: "We recommend that services use the <generatePublisherEvidence> element to improve startup performance. Using this element can also help avoid delays that can cause a time-out and the cancellation of the service startup." The same article explains that most asp.net/.net applications don't need to verify the signatures (certificates, and therefore CRLs): "The common language runtime (CLR) tries to verify the Authenticode signature at load time to create Publisher evidence for the assembly. However, by default, most applications do not need Publisher evidence. Standard CAS policy does not rely on the PublisherMembershipCondition. You should avoid the unnecessary startup cost associated with verifying the publisher signature unless your application executes on a computer with custom CAS policy, or is intending to satisfy demands for PublisherIdentityPermission in a partial-trust environment. (Demands for identity permissions always succeed in a full-trust environment.)"
The same article also clarifies, "[i]n the .NET Framework 4 and later, this element has no effect on assembly load times." This suggests to me that this option is more important for SharePoint 2010 (which uses .net 2.0/3.5) than for SharePoint 2013 (which uses .net 4.0/4.5).
The fact that it is recommended for aspnet.config and for owstimer.exe.config by a SharePoint Escalation Engineer here is also encouraging.
The easy part is modifying the config file with one simple, short line:
<configuration>
<runtime>
<generatePublisherEvidence enabled="false"/>
</runtime>
</configuration>
The not-so-easy part is deciding config file(s) to add it to? A web.config file? If so, which web.config file? (There are many to choose from!) What about adding it to an owstimer.exe.config file? (Often overlooked.) What about other .net applications and services that might also run on a SharePoint server? Do you need to create a config file named <ApplicationName>.exe.config and add the line to it? I like the fact that you've got great granularity here. But that might increase the odds of setting it at the less than optimal place. If you want to set it for some specific process or specific web application, you can. And if you want to set it at a "global level" or "root level" instead you can do that. But even at the root level you have some options to wade through. Which of these files should you edit?
C:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet.config
C:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG\machine.config
C:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG\web.config
C:\Windows\Microsoft.NET\Framework64\v2.0.50727\aspnet.config
C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG\machine.config
C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG\web.config
C:\Windows\Microsoft.NET\Framework\v3.5\aspnet.config
C:\Windows\Microsoft.NET\Framework\v3.5\CONFIG\machine.config
C:\Windows\Microsoft.NET\Framework\v3.5\CONFIG\web.config
C:\Windows\Microsoft.NET\Framework64\v3.5\aspnet.config
C:\Windows\Microsoft.NET\Framework64\v3.5\CONFIG\machine.config
C:\Windows\Microsoft.NET\Framework64\v3.5\CONFIG\web.config
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet.config
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\web.config
C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG\web.config
C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\CONFIG\web.config
Some bloggers go for the aspnet.config file (ref.1, ref.2) recommends making the change at the aspnet.config file. This would affect all the web application and websites that use asp.net. It doesn't affect any process that uses CLR/.Net. Setting it at the machine.config level would disable it not just for asp.net but all .net/managed code running on the server.
One fine point here is that SharePoint 2010 uses 2.0 and 3.5. So you might want to edit the config file of your choice in both the 2.0 and 3.5 folders. It might be a good idea to do it in the x86 and x64 folders too.
SharePoint 2013 uses .net 4.0. So there is probably no need to make any changes in the 2.0 or 3.5 folders.
If you decide to set it at "the hive level" (14\CONFIG and/or 15\CONFIG) you might expect the web.config in the 14 hive to affect the site collections running in 2010 mode (which is possible on a SP2013 server if you upgraded the database from SP2010 but didn't upgrade the site collection) while the web.config in the 15 hive should affect site collections running in 2013 mode.
It's also good to keep in mind that that disabling the generation of publisher evidence in a web.config file or aspnet.config file won't affect Owstimer.exe. For SharePoint servers running owstimer.exe, you may want to consider creating an owstimer.exe.config file. (Ref.)
Option 4 – Export the SharePoint Root Authority certificate of the farm and import it into the Trusted Root Certification Authorities store of each SP server in the Farm
This option comes from "Workaround 1" in http://support.microsoft.com/kb/2625048 ("Site slowness because of SharePoint STS certificate CRL Checking"). It is also recommended as workaround 1 in http://support.microsoft.com/kb/2672574 (SharePoint 2010 – Performance issue on external BDC lists).
I'm a big fan of this option. I like this option because it doesn't seem controversial to me that SharePoint servers in a farm should trust this certificate. I'm a little surprised that this doesn't happen the moment a server is joined to the farm. When a SharePoint farm is created, a root certificate is created for the servers in that farm to use for internal communication. The validity of that certificate is checked frequently. If this certificate is found in the local certificate store of the sharepoint server, trust is assured and communication goes on like it should. If the certificate is needed and it is not in the local store of the server, the server tries to validate the certificate over the network. This can cause unwanted delays.
Another thing I like about this option is that it doesn't disable any CRL checks. It's less controversial from a security standpoint than some of the other options then. But it's also not going to cover as wide a variety of problems as other options may give.
Option 5 - Disable the automatic update of root certificates on the SharePoint Servers
Technically this option really has nothing to do with CRL checking or the lack thereof. But it does often come up in the discussion of CRL checking and possible causes for performance problems in SharePoint BDC, Infopath Services, or anything that relies on the Secure Store service. This is discussed in detail at http://support.microsoft.com/kb/2672574 (SharePoint 2010 – Performance issue on external BDC lists).
This option is found in several articles on the internet. Most interestingly it is found as Workaround 2 in http://support.microsoft.com/kb/2672574 and again as workaround 2 in http://support.microsoft.com/kb/2625048. Personally I prefer workaround 1 in both instances. But it's good to know what all the options are.
KB 2672574 gives the best hint as to why one should or shouldn't do this: "There should not be specific implications to SharePoint since we are using self-signed certs and manage them ourselves. The SharePoint certificates do have an expiry and we do have a health rule that watches for that IIRC and will warn the admin to update/re-roll them. The main aspect to think through is for "other" certificates used on the box (like SSL certificates, certificates to trust download packages or for SAFER policy etc) which are issues from certificates chained to those in the TRC store."
The gist of it is to launch gpedit.msc as admin on the SharePoint server(s). (This assumes that it's not all greyed out due to being controlled by Group Policy.) Drill down into Local Computer policy > Windows Settings > Security settings > Public Key Policies. Double-click the Certificate Path Validation Settings. In the Network retrieval tab, add a checkmark beside "Define the policy settings" and uncheck the option to "Automatically update certificates in the Microsoft Root Certificate Program (recommended)." Run gpupdate /force.
![]()
![]()
Option 6 - Change the registry key for Software Publishing "State" value to '0x00023e00'
One common workaround to attempt to disable CRL checking is to open the registry and set HKEY_USERS\S-1-5-20\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing "State" to dword:00023e00.
The steps are simple:
Start Registry Editor (Regedit.exe)
Navigate to HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing
On the left side pane look for State key and double click to open it
Change the Value data to 146432 Decimal or 0x00023e00 Hexadecimal
No reboot is needed.
When "State" is set to 00023e00, certificate revocation is turned OFF. When "State" is set to 00023c00, it is turned on.
I admit that this can help avoid some CRL checking problems. But I am not very favorable towards it because (1) it's not a best practice from a security standpoint, (2) it's not very granular or flexible, (3) it seems like it should guarantee to solve all CRL checking related problems but there are times when it does not solve them.
One of the possible reasons it may not solve all CRL related problems is that sometimes the HKEY_CURRENT_USER entry may not be adequate. It *may* also need to be applied to locations as well, such as:
- HKEY_USERS\S-1-5-18\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing
- HKEY_USERS\.DEFAULT \Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing
In theory, a Process Monitor trace should show whether these keys are being touched or not. HKCU works for the "current user" but may not have anything to do with the service accounts that your application pools, for example, are running under.
Option 7 - Download and Import the fresh CRLs from crl.microsoft.com every month
If your servers are unable to reach out to http://crl.microsoft.com due to strict security policies but you also want to continue checking CRLs, you can.
You can download copies of the lists simply by browsing to these URLs:
There could be other CRLs that you might need. This is where CAPI logging comes in handy. The CAPI2 logs should show you which processes are reaching out to specific CLRs. Example:
![]()
But then you need to put them somewhere.
You can use certutil (already on your server) to import those CRLs to the local CRL store of each and every SharePoint Server.
The commands are simple:
Certutil –addstore CA CodeSignPCA.crl
Certutil –addstore CA CodeSignPCA2.crl
Certutil –addstore CA microsoftrootcert.crl
Certutil –addstore CA CSPCA.crl
Certutil –addstore CA MicCodSigPCA_08-31-2010.crl
Similarly, but as an alternative, you could place the downloaded .crl files on an IIS website, for example, that is accessible to the SharePoint servers, has directory browsing enabled, that is listening on port 80, and has a folder structure of pki/crl/products. And then you could create either an internal dns entry (or several hostfile entries) that resolve crl.microsoft.com to the IP address of the web server. The idea here of course is that whenever a server reaches out to http://crl.microsoft.com/pki/crl/*, they end up getting the CRLs from what amounts to an internal mirror.
So why don't more people do this? There is a catch here. Those CRLs are set to expire every two months (?) or so. They're always getting replaced by fresh CRLs. And if a certificate were revoked, the CRL might get replaced sooner than the two month expiration date. Even though it's easy to download the CRLs and import them, it's not something an Admin is going to want to do every two months. It is something that could be scripted, but that's a hassle too.
Option 8 - Uncheck "Check for publisher's certificate revocation" in Internet Explorer's Internet Options
Many have used the simple option of opening Internet Explorer on the servers, expanding the TOOLS menu, selecting Internet Options, selecting the Advanced tab, and unchecking the box beside "Check for publisher's certificate revocation."
![]()
(Some also might tempt the reader to also uncheck "Check for server certificate revocation.")
The main appeal to this is how simple it is. And it seems to work for solving many CRL checking problems. It disables CRL checking for pretty much everything on the server. It was especially popular in the days of ASP.net 1.1 and ASP.net 2.0 (before the ability to disable generatePublisherEvidence was added to 2.0).
I unchecked "Check for publisher's certificate revocation" on a test machine while running Procmon and saw that the majority of the changes made by unchecking this revolved around keys in the HKLM\SOFTWARE\Microsoft\Internet Explorer\AdvancedOptions\CRYPTO\ area, there was also a change at HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing – STATE.
Here is a screenshot of the salient part of the procmon capture:
![]()
So ultimately this is the same as Option 6. But to minimize the number of registry changes, I'd recommend option 6 over this one.
Thanks for taking the time to read this. I invite you to leave comments that will help me improve this blog.
Sources
http://support.microsoft.com/kb/2625048 Site slowness because of SharePoint STS certificate CRL checking
http://msdn.microsoft.com/en-us/library/bb629393.aspx <generatePublisherEvidence> Element
http://blogs.msdn.com/b/spses/archive/2012/08/29/crl-verification-failure-sharepoint.aspx
http://blogs.msdn.com/b/paulking/archive/2011/04/29/bypassing-the-crl-check-on-clr-startup.aspx
http://blogs.msdn.com/b/pfedev/archive/2008/11/26/best-practice-generatepublisherevidence-in-aspnet-config.aspx Best Practice - <GeneratePublisherEvidence> in ASPNET.CONFIG
http://social.technet.microsoft.com/wiki/contents/articles/2303.understanding-access-to-microsoft-certificate-revocation-list.aspx Understanding Access to Microsoft Certificate Revocation List
http://blogs.msdn.com/b/tom/archive/2008/10/28/web-site-stops-responding-for-15-25-seconds.aspx Web Site Stops Responding for 15-25 seconds
http://blogs.msdn.com/b/tess/archive/2008/05/13/asp-net-hang-authenticode-signed-assemblies.aspx ASP.NET Hang: Authenticode signed assemblies
http://blogs.msdn.com/b/gregmcb/archive/2008/05/06/ssl-and-authenticode-causes-crl-lookups-if-your-machine-cannot-access-the-crl-for-verification.aspx SSL and Authenticode Causes CRL lookups if Your Machine Cannot Access the CRL for Verification
http://support.microsoft.com/kb/2672574 SharePoint 2010 – Performance issue on external BDC lists
http://blogs.msdn.com/b/sharepointdev/archive/2011/02/08/error-the-sandboxed-code-execution-request-was-refused-because-the-sandboxed-code-host-service-was-too-busy-to-handle-the-request.aspx
http://blogs.msdn.com/b/asiatech/archive/2011/04/26/asp-net-application-loads-slowly-the-first-time.aspx
http://blogs.msdn.com/b/mcsnoiwb/archive/2012/04/25/how-to-speed-up-sharepoint-management-shell-startup.aspx
http://blogs.msdn.com/b/gregmcb/archive/2008/05/06/ssl-and-authenticode-causes-crl-lookups-if-your-machine-cannot-access-the-crl-for-verification.aspx
http://support.microsoft.com/default.aspx?scid=kb;EN-US;915850
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/seccrypto/security/wintrustsetregpolicyflags.asp
http://blogs.msdn.com/b/andreal/archive/2008/07/19/wcf-service-startup-too-slow-have-you-thought-to-crl-check.aspx
http://support.microsoft.com/kb/936707/
http://msdn.microsoft.com/en-us/library/aa388197
http://msdn.microsoft.com/en-us/library/windows/desktop/aa387700(v=vs.85).aspx
http://www.pdfsharepoint.com/error-sandboxed-too-busy-handle-request/
http://www.microtom.net/microsoft-sharepoint-products/how-to-make-sharepoint-faster
http://bizbert.com/bizbert/2007/03/24/BizTalk+And+Certificate+Revocation+Lists+CRLs+Part+II.aspx
Additional keywords:
The revocation function was unable to check revocation because the revocation server was offline
