SP2010: 503s, HTTP Throttling, Threads Waiting

September 24, 2013, 6:43 am

≫ Next: Can't open a .xsn file | "X-Download-Options: noopen" | MIME "application/octet-stream"

Symptoms

A web application in a SharePoint 2010 farm becomes unresponsive (hangs) to the client's perspective under high, medium, or even low client load at random times.

Clients of the SharePoint sites see 503 and/or "server too busy" or "The server is busy now. Try again later," in their browsers.

Admins may see events in the application event log that center around http throttling. Examples:

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          [Date and Time]
Event ID:      8062
Task Category: Http Throttling
Level:         Critical
User:          FOO\SP_WebApps
Computer:      WFE002.win.foo.org
Description:
Http throttling on SharePoint - 443 stops because there is no heavy load detected now. 472 requests have been throttled during the throttling period.

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          [Date and Time]
Event ID:      8032
Task Category: Http Throttling
Level:         Critical
User:          FOO\SP_WebApps
Computer:      WFE002.win.foo.org
Description:
Http throttling starts because a heavy load was detected on SharePoint - 443. The excessive performance counters include: WFE002.win.foo.org
\ASP.NET\Requests Current: 634.04

The c:\windows\system32\logfiles\httperr\httperr*.log show shows many "connection_dropped" entries followed by many "connection_abandoned_by_reqQueue" for the appPool corresponding to the unresponsive Web App. Examples:

[Date and Time] 172.30.122.196 58092 10.51.2.80 443 HTTP/1.1 GET /dist/5e/src/oruit/Team+Documents/Forms/AllItems.aspx - 1512381070 Connection_Dropped SharePoint+-+443
[Date and Time] 172.30.122.196 50060 10.51.2.80 443 HTTP/1.1 GET /nit/FOO/security/FSM/Documents/Forms/AllItems.aspx - 1512381070 Connection_Dropped SharePoint+-+443
[Date and Time] 10.110.77.146 64060 10.51.2.80 443 HTTP/1.1 HEAD /nit/FOO/OneNote/Open+Notebook.onetoc2 - 1512381070 Connection_Abandoned_By_ReqQueue SharePoint+-+443
[Date and Time]172.30.122.196 59270 10.51.2.80 443 HTTP/1.1 GET /nit/FOO/ins/Lists/MXTraining/overview.aspx - 1512381070 Connection_Abandoned_By_ReqQueue SharePoint+-+443
[Date and Time]172.30.122.196 61391 10.51.2.80 443 HTTP/1.1 HEAD /nit/FOO/Projects/Documents/BluePrint+Draft+2013.docx - 1512381070 Connection_Abandoned_By_ReqQueue SharePoint+-+443

A "hang dump" of the w3wp.exe process (made with either taskmanager's "Create user dump" option or with DebugDiag 1.2) shows a very high percentage of worker threads (example: 83.58% of threads blocked [499 threads]) waiting and "calling an ISAPI Extension OWSSVR." A debugdiag 1.2 crash/hang analysis script of such a hang dump may show, as an example, the following type of call stacks:

The following threads in w3wp.exe__Application Pool (6.13.2011 6.01.13 PM)__PID__13912__Date__09_13_2013__Time_05_05_16PM__920__Manual Dump.dmp are calling an ISAPI Extension OWSSVR (C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\ISAPI\OWSSVR.DLL.)

( 15 16 17 18 19 20 21 22 39 45 47 55 56 57 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 207 208 209 210 212 213 )

77.21% of threads blocked (166 threads)

Entry point msvcr90!endthreadex+64

Create time [Date and Time]

Time spent in user mode 0 Days 00:00:06.140

Time spent in kernel mode 0 Days 00:00:01.484

This thread is calling an ISAPI Extension OWSSVR

Full Call Stack

Function

ntdll!ZwDelayExecution+a

KERNELBASE!SleepEx+ab

ONETUTIL!Vistream::error+191

ONETUTIL!VonetLocks::CReaderWriterLock3::_LockSpin+e3

ONETUTIL!CLKRLinearHashTable::_ReadOrWriteLock+55

ONETUTIL!CLKRLinearHashTable::_FindKey+36

ONETUTIL!CLKRHashTable::FindKey+88

STSWEL!VglobalAuditStore::ensureDatabaseQueue+d0

STSWEL!VglobalAuditStore::addAuditEntry+d8

STSWEL!auditSecurityScopeEvent+27d

STSWEL!VdocumentStore::httpGetDocument+36cf

STSWEL!VhttpManager::loadFileCore+7ef

STSWEL!VhttpManager::loadFile+f7

STSWEL!VhttpManager::handleNormalFetch+73f

STSWEL!VhttpManager::handleCommonFetch+37

STSWEL!VhttpManager::handlePOST+18

STSWEL!VhttpManager::dispatchHttpRequest+2d6

OWSSVR!MsoFAssertSzTagProcVar+6bdf1

OWSSVR!MsoFAssertSzTagProcVar+5957f

ONETUTIL!Vframework::doMain+a7

OWSSVR!MsoFAssertSzTagProcVar+856

ONETUTIL!COWSThreadWithHeap::WalkHeap+1dc

ONETUTIL!COWSThreadWithHeap::Uninitialize+66a

msvcr90!endthreadex+47

msvcr90!endthreadex+e8

kernel32!BaseThreadInitThunk+d

ntdll!RtlUserThreadStart+1d

All the other waiting ("blocked") threads will look identical to that. Note especially the ONETUTIL!VonetLocks::CReaderWriterLock3::_LockSpin+e3 and the STSWEL!VglobalAuditStore::addAuditEntry.

Third party monitoring tools may flag "IIS: ASP Application restarts" and "IIS: ASP Request Execution Time."

Resolution

Consider applying SharePoint 2010's Service Pack 2 *AND* the August 2013 Cumulative Update (or higher). Link: http://technet.microsoft.com/en-us/sharepoint/ff800847.aspx

More Information

Changing the HTTP Throttling threshold with PowerShell on the WFE from default of 500 to 750 is probably not going to ultimately help. But this may be the first reaction some Admins consider trying to minimize the symptoms. Requests are queuing because the worker threads are all waiting. Increasing the throttling threshold probably won't do anything to avoid that problem. The problem is not too many requests to process; it is the inability to process the requests.

↧

Can't open a .xsn file | "X-Download-Options: noopen" | MIME "application/octet-stream"

October 5, 2013, 12:41 pm

≫ Next: Steps to Catch a Simple “Crash Dump” of a Crashing Process

≪ Previous: SP2010: 503s, HTTP Throttling, Threads Waiting

After upgrading MOSS 2007 content databases to SP2010, an InfoPath form template (template.xsn) that in the past could be either opened or saved was now only able to be saved, not opened.

Comparing Fiddler2 traces of the traffic for the .xsn file on the MOSS2007 and SP2010 farms showed that the key difference in the headers was this: “X-Download-Options: noopen.” Here's a great reference for that. SharePoint 2010 and 2013 are a bit more strict here than MOSS 2007 was.

Since the problematic header also showed clearly that that the content type was "application/octet-stream" we added the necessary MIME type with the following PowerShell commands:

$webApplication = Get-SPWebApplication "http:/yourwebapplicationurl"
$webApplication.AllowedInlineDownloadedMimeTypes.Add("application/octet-stream")
$webApplication.Update()

Additional keywords:

Do you want to save this file?

↧

Steps to Catch a Simple “Crash Dump” of a Crashing Process

November 12, 2013, 12:40 pm

≫ Next: Several Good Ways to Trigger a “Hang Dump” of an Unresponsive Process

≪ Previous: Can't open a .xsn file | "X-Download-Options: noopen" | MIME "application/octet-stream"

Summary

When a user-mode process (such as w3wp.exe, owstimer.exe, iexplore.exe, or any .exe) is crashing (such that the process disappears unexpectedly from task manager and reappears soon after with a different PID number), arguably the best way to begin to determine the root cause of the crash is to catch a crash dump as that process is crashing. This blog offers two ways to trigger the crash dump and suggests a way to begin interpreting the dump.

The Choice between DebugDiag and WER

I highly recommend the use of DebugDiag as the tool to use to automatically trigger a crash dump as the process crashes and as the tool to begin analyzing the crash dump. The tool is simply the most flexible and customizable way to get simple crash dumps and advanced crash dumps. But since I'm taking here about simple crash dumps, and since it's not uncommon for the change management process to move too slowly, and since some IT groups have very restrictive tool installation approval process for their servers, sometimes DebugDiag cannot be installed on a server where a process is crashing, I also am going to offer steps here to use Windows Error Reporting (WER) to automatically trigger the crash dump. Often WER is adequate to get the crash dump. So I'll start with WER. But if you have permission to install DebugDiag on your server, feel free to skip the WER steps.

How to Set up WER for a Crash Dump

WER is already there! WER is already part of your operating system (in Win2008 and higher) and needs only to be configured. You'll need to add a registry key.

As an example, to catch a total of two crash dumps of a crashing w3wp.exe process, you could use copy the following text into notepad, save it, and rename it to something like W3wpcrashdumps-to-C.reg.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\w3wp.exe]

"DumpFolder"=hex(2):63,00,3a,00,5c,00,64,00,75,00,6d,00,70,00,73,00,00,00

"DumpCount"=dword:00000002

"DumpType"=dword:00000002

This will make a total of 2 full user dumps (not mini-dumps but full dumps) and write the dumps to C:\dumps.

In the registry it will look like this:

But you may not want to write the dumps to the system partition. You can send it to another drive with a few GB of free space instead.

So, as an example, to send the dumps to the D:\ drive you could use something like this:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\w3wp.exe]

"DumpFolder"=hex(2):64,00,3a,00,5c,00,64,00,75,00,6d,00,70,00,73,00,00,00

"DumpCount"=dword:00000002

"DumpType"=dword:00000002

In the registry it will look like this:

As long as this registry key is there, expect dumps to be written every time any w3wp.exe crashes—up to a maximum of two dumps if DumpCount is set to 2. So when you've collected enough dumps, don't forget to either delete the w3wp.exe key from the registry.

Of course if the crashing process is something other than a w3wp.exe, you'd want to create a different key under LocalDumps. For example, if OWSTimer.exe was crashing, you'd create a key for OWSTimer.exe.

References: http://msdn.microsoft.com/en-us/library/bb787181.aspx

INSTALL DEBUGDIAG

Assuming you did not configure WER with the steps above, install Debug Diagnostic 2.0 on the server suffering with a crashing process. Download it from http://debugdiag.com.

When you launch debugdiag.msi it will give you an option to change the default installation path. If you prefer to not install tools to the system partition, be sure to select the Browse button in the earliest phase of the wizard.

LAUNCH DEBUGDIAG

Launch Debug Diagnostics 2.0 Collection from the list of programs.

If UAE is enabled on the server, you may need to use the "Run as Administrator" option to avoid errors during launch.

SET UP A VERY BASIC CRASH RULE

When asked which rule type to choose, keep the bullet beside "Crash" and click NEXT

For "Select Target Type" you have some great choices.

If you need to get a dump of a process like owstimer.exe or iexplore.exe or anything not related to IIS, you should probably select "A specific process."
If you have a w3wp.exe that is crashing and you know which Application Pool it is associated with, you should probably go with "A specific IIS web application pool."
If you have a w3wp.exe that is crashing and you don't know which AppPool it is associated with, you should probably go with "All IIS/Com+ related processes."

As an example, I'm selecting notepad.exe. It could be any other process.

Leave everything set to the default setting in the Advanced Configuration for now. Just click NEXT.

Continue accepting the default settings and clicking next…

If you prefer that the .dmp files to be written to some place other than the default, select browse and guide it to another folder. Generally speaking it's probably good to not write the dumps to the system drive, especially if you have limited freespace.

Click Next.

Activate the rule when you're ready for the tool to start monitoring the process for a crash. . .

Just click YES here.

Note how the status is set to active and the userdump count is set to 0. The userdump count should increase when a crash is detected.

You can also activate and deactivate the crash rule from this view.

WAIT

Feel free to log off the server while waiting for the crash to occur. Since debugdiag runs as a service, you do not have to be logged into the machine.

This tool will monitor the process, watching and waiting for a crash to occur. When the process begins to crash, the debugger will interrupt the process temporarily, freeze that process, write out everything in that process to a .dmp file, and then allow the process to crash and restart.

Preliminary Analysis of the Crash Dump

This step can be done from a server if (1) that server has outbound http access to the internet [specifically to Microsoft's public symbol servers] and (2) if you don't mind risking something that is fairly CPU intensive. It may make more sense to install Debug Diag 2.0 (or just the analysis piece) on a workstation that can access the internet and place the crash .dmp file on a share that it can reach.

Launch the Debug Diag Analysis program from the list of programs.

Place a checkmark beside CrashHangAnalysis. Don't select any other analysis rules.

Select "Add Data Files" and guide it to the crash dump.

Select START ANALYSIS and wait for results to display.

Zipping the Dump

If you'd like to zip the dump file up in preparation to upload to an engineer at Microsoft, here is a good way to do it.

Expand the debugdiag Tools menu, select Advanced Data Collection, select Create Full Cabinet file. This should both collect and compress the event logs, the .net config files, the dump files, and more into one convenient .cab file.

You can locate the .cab file by clicking the icon of the manila file folder.

Making the Crash Rule More Sensitive

If this method above does not catch a crash and create a dump file when the crash occurs, you can edit the existing crash rule and add a breakpoint to make it more sensitive with the steps below.

Keep in mind, however, that this method may create a dump whenever a process is recycled or is otherwise gracefully shut down rather than when it actually crashes. So if, for example, you recycle your application pools at 2am every morning, and your crash rule creates a dump of the w3wp.exe at 2am, you might want to ignore and delete that dmp file.

What if the dump shows heap corruption?

If the dump analysis shows signs of heap corruption as the cause of the crash, you might need to adjust your crash rule to enable gflags (as seen below) and wait for another crash. Gflags can only be enabled if during the creation of the crash rule you selected "specific process." Also enabling gflags can have a profound impact on server performance. This is some advanced work that is probably best done with the guidance of a support engineer at Microsoft. The debugging can be challenging and there is also a good chance that there may be need to switch to another tool (such as iDNA/TTTracer).

Additional search keywords

919789 How to use the Debug Diagnostics tool to troubleshoot an IIS process that stops unexpectedly

http://support.microsoft.com/default.aspx?scid=kb;EN-US;919789

921464 How to use the Debug Diagnostics 1.0 tool to analyze a memory dump file

http://support.microsoft.com/default.aspx?scid=kb;EN-US;921464

Event Source: W3SVC

Event ID: 1009

Description: A process serving application pool 'xyz' terminated unexpectedly. The process id was '1234'. The process exit code was '0xc0000005'.

Event Source: W3SVC

Event ID: 1011

Description: A process serving application pool 'xyz' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '1234'.

Event Source: Application Popup

Event ID: 26

Description: Application popup: inetinfo.exe - Application Error : The instruction at "0x01b2dc59" referenced memory at "0x00000008". The memory could not be "read". Click on OK to terminate the program

Source: Microsoft-Windows-WAS

Event ID: 5011

Description: A process serving application pool 'XYZ' suffered a fatal communication error with the Windows Process Activation Service. The process id was '2780'. The data field contains the error number.

Source: .NET Runtime

Event ID: 1027

Description: Application: w3wp.exe Framework Version: v4.0.30319

Description: The process was terminated due to stack overflow.

Event ID: 1000

Description: Faulting application w3wp.exe, version …, time stamp …., faulting module xyz.dll, version …., time stamp …., exception code ….., fault offset ….., process id …., application start time …..

Source: Microsoft-Windows-WAS

Event ID: 5009

Description: A process serving application pool 'XYZ' terminated unexpectedly. The process id was '1234'. The process exit code was 'l'.

Event Source: W3SVC

Event ID: 1011

Description: A process serving application pool 'XYZ' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '1234'. The data field contains the error number.

Event ID: 1000

Description: Faulting application w3wp.exe, version 7.0.6001.18000, time stamp 0x47919ed8, faulting module kernel32.dll, version 6.0.6001.18000, time stamp 0x4791ada5, exception code 0xe0434f4d, fault offset 0x000000000002649d, process id 0x1e90, application start time 0x01ca00762fbfb6f3.

Access violation

Second-chance exception

The SharePoint 2010 Timer service terminated unexpectedly. It has done this 696 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service. (Event ID: 7031, Source: Service Control Manager)

The SharePoint 2010 Timer service terminated with service-specific error %%-2147467259. (Event ID:7024, Source: Service Control Manager)

adplus.vbs –crash

"error while attaching to process via the DbgSvc service. ReloadControlScriptFailed. Could not open handle to control script shared memory mutex.")

↧

Several Good Ways to Trigger a “Hang Dump” of an Unresponsive Process

November 14, 2013, 9:15 pm

≫ Next: Steps to Log all .net Exceptions to a Log file with Debug Diag 2.0

≪ Previous: Steps to Catch a Simple “Crash Dump” of a Crashing Process

Introduction and Summary

Imagine a traffic jam where you can see cars and trucks backed up for miles. You can see the symptoms and know there is a problem. You know things have slowed to a crawl or to a stop. But that's all you can see from your vantage point. You can't see the choke point. You can't see what the real cause is of the traffic jam. All you can see are the lines of backed up cars. And it's your job to fix it! What do you do?

You can make guesses. "A car probably ran into another car and emergency response crews are probably slowing traffic down." That sounds reasonable.

But you have no idea which cars collided if indeed there was a collision. You have no idea if the emergency response crews are really there or not and if they're going to take care of the problem or not.

If only you could get a good glimpse of the root cause, right? Even just a good momentary snapshot of a few traffic cameras in the area would help unravel the mystery.

If you could see the snapshot from a webcam you might be surprised at what you find. . .

A parade of elephants?

That one snapshot can increase your understanding of the root cause of the problem can begin and you're far closer to finding a solution to the problem.

This is what I like about making "hang dumps" of user-mode processes when they're unhealthy. With a good memory dump made at the right time you can see what the various worker threads are doing. Some are doing work, some are waiting in a good way, one is causing a traffic jam, and several others are blocked by the jam. And with that snapshot of who is doing what, you get a chance to see what is causing the problem. You start to be able to see what is smoke and what is fire.

When a website hosted in IIS becomes inordinately slow or unresponsive, I am quick to recommend creating a dump of the hung w3wp.exe process.

When SharePoint workflows become unresponsive, I am quick to recommend creating a dump of the owstimer.exe process to begin to pinpoint the root cause.

When any process to become "hung," "jammed," "blocked," or otherwise unresponsive, I'm quick to trip a dump while the hang is occurring.

This blogpost recommends several good ways to create a "hang dump" of a process. First we'll see how to dump out a process with Task Manager. Then we'll see three ways to get the hang dump with Debug Diag 2.0.

Task Manager

Perhaps the quickest and easiest way on a Windows 2008, Windows 2008 R2, Windows 7, Windows 8, or Windows 2012 computer is to use task manager. If Outlook were unresponsive, for example, I would open Task Manager, select the Processes tab, find the process listed in the list, right-click it, select "Create Dump File," wait for the dump file to be written, and then click the option of End Task to kill the hung process. The beauty of this method is that Task Manager is already there. You don't have to install any tools. The memory dump it creates should be just as viable as dumps made by other tools.

Open task manager (one good way to do this is to press keyboard keys "Ctrl" + "Shift" + "Esc" simultaneously) and select the Processes tab.

If you know which process is unhealthy, you can locate it in the list of processes, right-click it, and select CREATE DUMP FILE.

Yeah, it can be that easy! Just make sure you dump it while it is unhealthy/unresponsive.

One common dilemma is that if you have multiple processes with the same name (often the case with w3wp.exe's) you may not know which one to dump. The main downside to using Taskman to get the dump is that sometimes it's difficult to figure out which process to dump. If you have a web server or SharePoint Web Front End server that has multiple Application Pools, sometimes it takes a bit of detective work to figure out which w3wp.exe corresponds to which application pool.

If you're not sure which one needs to be dumped, you could just methodically right-click and dump every one of the processes with the same process name.

Or maybe you can tell by the "user name" that the process runs as which w3wp.exe corresponds to which application pool. But that's not always going to work. Sometimes several App Pools might use the same user ID.

Or maybe you can tell by the size of the memory (private bytes) footprint. It's not uncommon for the unhealthy w3wp.exe to be the largest. But that's not always the case.

If the problem is due to high cpu, it may be quite obvious in taskman which process to dump. But often an unresponsive process has very low cpu. So cpu usage isn't always a good indicator.

One good way to figure out which w3wp.exe corresponds to which application pool is to run an appcmd command. Open a command prompt on the web server and run the following command to see which Application Pools line up with which PID numbers:

%windir%\system32\inetsrv\appcmd.exe list wp

Then of course I have to hunt around in Task Manager to find out how to view the PID numbers (process identifier) of the processes. But at least I can still quickly know which w3wp.exe is the one to dump. If you know which website is unresponsive, and you know which application pool that website is assigned to, and you know the pid number of that appPool, you should be able to figure out which PID to dump in Task Manager.

Another possible downside of using TaskMan for creating your dumps is that by default it will write the dump to a path like C:\Users\UserName\AppData\Local\Temp\ProcessName.DMP. That's not the easiest place to find. And maybe you don't want large dump files on your server's C drive.

Despite these two small drawbacks, all in all it is difficult to beat the convenience of the Taskman hang dump.

DebugDiag 2.0 – Installation and Launch

To have additional flexibility in making hang dumps consider installing Debug Diagnostics 2.0 on the server suffering with an unresonsive process. Download it from http://debugdiag.com.

LAUNCH DEBUGDIAG

Launch Debug Diagnostics 2.0 Collection from the list of programs.

If UAE is enabled on the server, you may need to use the "Run as Administrator" option to avoid errors during launch.

If you installed the tool to your system drive but would prefer that the memory dumps go to a different drive, expand the Tools menu and select Options and Settings.

Change the manual user dump save folder path and click OK

DebugDiag – Option 1: Create Full User Dump from the Processes Tab

After debugdiag launches, click the CANCEL button on the Select Rule Type window

Switch to the Processes tab in DebugDiag

Sort by process name or by whatever column makes the most sense.

For IIS web app pool problems, note especially the column for Web Application Pool name. With this cool feature there is no need to run appcmd list wp.

To trigger the hang dump, right-click the process you wish to dump and select "Create Full Userdump."

Be sure not to select MINI userdump.

DebugDiag – Option 2: Dump ALL the IIS processes from the Tools menu

If all you know is that one of your IIS websites has a problem and you don't have time to figure out which application pool to focus on,

you might consider expanding the Tools menu

and selecting CREATE IIS/COM+ HANG DUMP.

This will create hang dumps of all the IIS related processes.

DebugDiag – Option 3: Set up a performance rule to detect the problem and automagically dump the process

Sometimes the hangs happen at 4 AM and no one is around to respond to the hang.

In that case, you might want to create a performance rule to detect the hang and automatically make the dump.

From the rules tab, select Add Rule, place a bullet beside Performance, and click NEXT.

I'm not going to go through these scenarios here, but check out how cool these options are.

Your "performance rule" can predicate actions (such as triggering a hang dump) based on what perf counters say or based on checking a URL for responsiveness.

Definitely a cool option to have!

Preliminary Analysis of the Hang Dump

Launch the Debug Diag Analysis program from the list of programs.

Place a checkmark beside CrashHangAnalysis. Don't select any other analysis rules.

Select "Add Data Files" and guide it to the crash dump.

Select START ANALYSIS and wait for results to display.

Zipping the Dump

If you'd like to zip the dump file up in preparation to upload to an engineer at Microsoft, here is a good way to do it.

You can locate the .cab file by clicking the icon of the manila file folder.

PROCDUMP

Some people really like Procdump. I haven't used it much myself but I thought it was worthy of mention.

http://technet.microsoft.com/en-us/sysinternals/dd996900

ProcDump v6.0

By Mark Russinovich

Introduction

ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike. ProcDump also includes hung window monitoring (using the same definition of a window hang that Windows and Task Manager use), unhandled exception monitoring and can generate dumps based on the values of system performance counters. It also can serve as a general process dump utility that you can embed in other scripts.

↧

Steps to Log all .net Exceptions to a Log file with Debug Diag 2.0

November 19, 2013, 3:03 pm

≫ Next: Troubleshooting the vague “Cannot complete this action” owssvr.dll error

≪ Previous: Several Good Ways to Trigger a “Hang Dump” of an Unresponsive Process

Sometimes it's good to know what exceptions a process like w3wp.exe is throwing. ASP.net is pretty good about making its complaints known in the form of exceptions. Many or perhaps most exceptions can be ignored. But if you're trying to troubleshoot a mysterious problem that isn't leaving clues in the event logs, this may be a good way to begin to unravel the problem. Sometimes you can see them logged in the Application event log—but usually not. Sometimes you can see them in the ULS logs of SharePoint servers—but usually not. You could get the developer can add some additional instrumentation to the code to log the exceptions to a log file. Or you might use Todd Carter's SNAP tool to see what .net exceptions are being thrown. Or you can, as I'll suggest here, try to use Debug Diagnostics 2.0 to log all .net exceptions to its log file while reproducing a problem.

Install DebugDiag 2.0 on the server. Location: http://debugdiag.com

If you can't get permission to install it with the .msi, consider the xcopy option here.

Launch DebugDiag 2.0 Collection

Confirm that "Crash" is selected as the rule type and click NEXT

Player that I am in the SharePoint universe, I tend to focus on a specific w3wp.exe that corresponds to a specific IIS Application Pool.

Assuming you know which Application Pool to focus on, select "A specific IIS web application pool."

Select the Application Pool in focus and click NEXT

For good measure I like to log some of the native code exceptions (non-.net) as well as the managed/.net exceptions.

SharePoint uses both native and managed code. This will help give some insight into the native side.

So change action type to "log stack trace" and change action limit to a high number (like 9999 or 999999).

Click the EXCEPTIONS button

Click the ADD EXCEPTION button

On the left side, select CLR (.NET) 1.0 – 3.5 Exception

On the right side, set Action Type to "Log Stack Trace" and Action Limit to a high number (like 9999 or 99999)

Click OK

Click the SAVE&CLOSE button

Click NEXT

Feel free to change the Rule Name to something like "Exception logging for Application Pool XYZ" if you please.

Feel free to change the rule output location.

Click NEXT

Place a bullet beside "Do not activate the rule at this time" and click FINISH.

If you want to have a "baseline" to have a record of which exceptions are "normal," right-click the rule, activate it, generate some activity on the website without duplicating the problem you're hoping to troubleshoot, deactivate the rule, and then proceed with the next steps.

It's probably a good idea to get additional debug information added to the log file.

To do so, expand the tools menu, select Options and Settings

Place a checkmark beside "Enable raw debugger logs. Includes debug output and engine messages."

Click OK

Get everything as ready as possible to reproduce the problem. If possible, get to the point where you are one simple mouse-click away from reproducing your problem.

When you're ready to reproduce the problem, right-click your rule and activate it.

Reproduce your problem quickly

After you see your symptom or error message in the browser, deactivate the rule.

Find your new log files by clicking the icon of the manila folder. This will open the folder C:\Program Files\DebugDiag\Logs

The file you're looking for may have a name that looks something like this:

w3wp__DefaultAppPool__PID__4840__Date__07_19_2013__Time_04_02_18PM__81__Log.txt

If your application is throwing several exceptions every second, doing this type of logging can really bog down the machine. Beware. And note these cool new features in 2.0 from the Tools > Options and Settings > Crash Rule Preferences tab. . .

More details here: http://blogs.msdn.com/b/debugdiag/archive/2013/10/02/did-you-know.aspx

Additional keywords:

System.InvalidCastException

System.OutOfMemoryException

System.IO.FileNotFoundException

System.NullReferenceException

System.CannotUnloadAppDomain

System.Threading.SynchronizationLock

System.InvalidOperationException

System.OverflowException

System.Data.SqlClient.SqlExeption

System.Data.VersionNotFoundException

System.ArgumentException

System.ExecutionEngineException

System.ArgumentOutOfRangeException

System.IndexOutOfRangeException

System.Net.WebException

System.Security.SecurityException

↧

Troubleshooting the vague “Cannot complete this action” owssvr.dll error

November 27, 2013, 12:15 pm

≫ Next: How to set up failed request tracing for Slow Pages in IIS 7

≪ Previous: Steps to Log all .net Exceptions to a Log file with Debug Diag 2.0

So what can you possibly do when you try to do something simple in SharePoint, like editing a view, and you see the following error in your browser?

Error

Cannot complete this action.

Please try again.

Troubleshoot issues with Microsoft SharePoint Foundation.

The only other clue you have is the presence of "/_vti_bin/owssvr.dll?CS=65001" in your address bar.

There is no correlation ID to focus on. And no matter how long you comb through the ULS logs, there are just no clues whatsoever.

What do you do?

The first thing I did in this scenario was to try to amp up the level of verbosity in ULS logging to verboseEX. Steps:

1. During a server maintenance window, log onto the central admin server as a Farm Admin

2. Open the SharePoint 2010 Management Shell (or PowerShell)

3. Run the following commands:

a. New-SPLogFile [Note: this starts a new ULS log]

b. Set-SPLogLevel -TraceSeverity VerboseEX [Note: this sets all logging to super high levels]

4. Note the time on the server

5. Begin to reproduce the problem

6. After problem has been sufficiently reproduced run the following commands

a. New-SPLogFile

b. Clear-SPLogLevel [Note: This sets the logging level back to normal/default]

7. Create a new folder to handle ULS logs (example here: D:\Logs)

8. Run this command: Merge-SPLogFile -Path "C:\Logs\FarmMergedLog.log" -Overwrite -StartTime "10/11/2013 16:00" – EndTime "10/11/2013 16:15"

· NOTE: path, date, and times may vary

· NOTE: this command is supposed to collect the ULS logs from every server in the farm and merge them into one biiiiiiiiig logfile. It's important to give it a specific time range. Details.

But to my dismay, even verboseEx level ULS logs were totally disappointing. No clues to be found there. Zero.

When ULS logs disappoint me, I'm usually pretty quick to turn to DebugDiag. So we installed DebugDiag 2.0 on the WFE (this farm fortunately only had one WFE and that makes it easy to guess which WFE your traffic is going to) and began to set up the crash rule to log all exceptions per my steps here. The log file that DebugDiag generated while we reproduced the problem showed something unexpected in one of the stack traces of a thread that was throwing a first chance exception.

Child-SP RetAddr Call Site

000000002c9bddb0 000007ff0024c51d Microsoft.SharePoint.SPListCollection.GetListByName(System.String, Boolean)

000000002c9bde30 000007ff0024c0d5 .(Microsoft.SharePoint.SPWeb)

000000002c9bde80 000007fef389ea69 .()

000000002c9bdf10 000007fef374a55c Microsoft.SharePoint.SPSecurity+<>c__DisplayClass4.<RunWithElevatedPrivileges>b__2()

000000002c9bdf90 000007fef36189a2 Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated)

000000002c9bdfe0 000007fef3618a9d Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(System.Threading.WaitCallback, System.Object)

000000002c9be080 000007ff0024986b Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated)

000000002c9be0e0 000007ff00249698 .()

000000002c9be150 000007ff00249549 ..ctor()

000000002c9be1a0 000007ff002490bd .()

000000002c9be210 000007feee7059b0 SusQtech.Products.UrlManager.HttpModule.(System.Object, System.EventArgs)

000000002c9be290 000007feee6f671b System.Web.HttpApplication+SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

000000002c9be2f0 000007feeede3971 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

000000002c9be390 000007feeedd43b2 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)

000000002c9be520 000007feeedb5df9 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)

000000002c9be570 000007feeeeddde1 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)

000000002c9be690 000007feeeedd9ab System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)

000000002c9be810 000007feeeedcc14 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)

000000002c9be870 000007fef833b1fb DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)

We retracted the SusQTech URL Manager solutions, ran IIS reset, and confirmed that the problem went away. I recommended to the server admin that he contact support from the vendor and see if they had newer versions or private fixes that might avoid the problem.

Although we didn't try it this time, we might have been able to find the same information (see which httpmodules were at play here anyway) with some IIS level failed request tracing.

↧

How to set up failed request tracing for Slow Pages in IIS 7

November 27, 2013, 12:28 pm

≫ Next: IIS Health Checking can Prevent the Creation of a W3WP.exe Userdump

≪ Previous: Troubleshooting the vague “Cannot complete this action” owssvr.dll error

These steps can be helpful for troubleshooting web pages that serve slowly.

Open the IIS manager, select the site that is having problems, select Failed Request Tracing from the far right column.

Set a checkmark beside ENABLE

Select a path for the log file to be written to. You may need to ensure that the account the application pool uses has permission to write to this location.

I don't usually care for more than 50 trace file.

Select the Failed Request Tracing Rules icon from the center column.

Typically I select "All content."

Setting a range of 200 to 500 will catch pretty much everything that gets past http.sys and to an application pool.

Setting "time taken" to 5 seconds makes it so that it will only log requests that take five or more seconds to serve.

If the site is a SharePoint site, don't forget about Developer Dashboard.

↧

IIS Health Checking can Prevent the Creation of a W3WP.exe Userdump

December 18, 2013, 12:20 pm

≫ Next: An Arcane Error with MOSS 2007’s Recycle Bin

≪ Previous: How to set up failed request tracing for Slow Pages in IIS 7

Problem

Sometimes when we're trying create a usermode dump of a w3wp.exe process something sabotages the attempt. The dmp file isn't successfully created and the dump-triggering tool balks.

When using Taskmanager to try to create the dump, the error is:

Dump Process

The operation could not be completed.

Only part of a ReadProcessMemory or WriteProcessMemory request was completed.

When using DebugDiag 2.0 to try to create the dump, the error is:

Debug Diagnostic Tool

Could not create dump file *.dmp for process id *.

GetLastError returns 0x8007012B.

This scenario is most common when the w3wp.exe is large (in terms of virtual bytes) and the web server may be a bit taxed on either CPU or memory. The problem may not happen when trying to dump out small w3wp.exe processes.

Root Cause

When you're making a memory dump of a usermode process, everything in the process gets frozen in time while the dmp file is written out. While everything (all the threads) is frozen inside the w3wp.exe, IIS health checking is "pinging" the Application Pool to look for problems like unresponsiveness. If the w3wp.exe seems unresponsive, the Windows Process Application Service (WAS) will recycle that Application Pool.

The sign of this happening is in the System Event Log. Look for WAS as the source of the event and 5010 as the event ID. Event ID 5010 will say something like, "A process serving application pool . . . failed to respond to a ping. The process id was . . ."

Workaround

By default, WAS will ping the appPool every 30 seconds and gives it 90 seconds to respond. If the worker process can't respond, WAS will recycle the Application Pool. The unresponsive w3wp.exe disappears and a fresh one takes its place.

There are a few options to consider.

You can try dumping the w3wp.exe sooner while its smaller. If you're troubleshooting a memory leak however, you may want to dump the process while its big.
You can set Ping to disabled for the AppPool in the IIS manager. Wait for a server maintenance window, open the INETMGR, drill down to the Application Pools, find and select the AppPool that you're trying to get a dump of, select ADVANCED SETTINGS, scroll down to PING ENABLED, and change from TRUE to FALSE.
Or change the Ping Maximum Response time to number higher than 90 and/or the Ping Period to some number higher than 30.

↧

An Arcane Error with MOSS 2007’s Recycle Bin

January 10, 2014, 12:59 pm

≫ Next: In case you are unable to fully delegate Search Administration in SP2013

≪ Previous: IIS Health Checking can Prevent the Creation of a W3WP.exe Userdump

Normally I wouldn't bother making a blog that probably only applies to MOSS 2007. But while solving for this error message . . .

Error

You must first restore the original list in order to restore the folder

Troubleshoot issues with Windows SharePoint Services.

. . . I was surprised at how there was just no information anywhere on this error anywhere. Figuring someone will run into this, I thought I'd fill the gap.

The problem and fix were ultimately quite simple. A site collection owner had deleted a document library, recreated the library manually, and then attempted to restore the documents and folders from the recycle bin. But he couldn't restore from the recycle bin. He got the error. The administrator then tried to restore folders or files from the recycle bin and got the same error message. ULS logs were not useful at all.

The problem was that we didn't know exactly what the site collection owner had done. We didn't know he had recreated the document library. So the Admin didn't know to look harder in the recycle bin for the doc lib. After deleting the new doclib he could restore the old doclib and proceed to restore the documents and folders.

↧

In case you are unable to fully delegate Search Administration in SP2013

February 17, 2014, 7:39 am

≫ Next: Three Ways to Automate a “Hang Dump” in DebugDiag 2.0

≪ Previous: An Arcane Error with MOSS 2007’s Recycle Bin

Problem:

When attempting to delegate SharePoint 2013 Search Administration to a small team/group, we were able to delegate everything except for the "Result Sources" and "Query Rules" pages/links, as seen in yellow highlight below.

Clicking either page resulted in the "Sorry, this site hasn't been shared with you" (Access Denied/ Access Required) message despite having done "all the right things."

If we added the delegated admin group to the farm administrators group, it all works of course. But we we're trying to taking it that far.

Solution:

This was resolved in the October 2013 Cumulative Update for SP2013 (and higher). Link: http://technet.microsoft.com/en-us/sharepoint/jj891062.aspx

↧

Three Ways to Automate a “Hang Dump” in DebugDiag 2.0

February 17, 2014, 9:22 am

≫ Next: Footnote Numbering Trick for Word 2013

≪ Previous: In case you are unable to fully delegate Search Administration in SP2013

Normally I prefer for the server admin to manually trigger a hang dump (see http://blogs.msdn.com/b/chaun/archive/2013/11/15/several-good-ways-to-trigger-a-hang-dump-of-an-unresponsive-process.aspx) when the process is clearly unresponsive and end-users are complaining. That way we have human intelligence working to ensure that the hang dump is made while the process is obviously unresponsive. Timing is important. There's no point in making a hang dump of a process that isn't hung up.

But there are times when it's just not feasible for the server admin to manually trigger a hang dump at a good time. For example, there may be times when the process becomes unresponsive in the middle of the night and there is no one around to report the problem to the help desk. Or maybe help desk just creates a ticket and the admin finds the ticket on the hang issue in the morning. Perhaps the process is still hanging around and still hung (unlikely but possible) or orphaned and the admin makes a memory dump of the process—but there's just no activity in the dump. Sometimes it is good to trigger the hang dump soon after the hang occurs rather than hours after the fact.

One of cool things about Debug Diag 2.0 (http://debugdiag.com) is that there are no less than three ways to automate the triggering of a "hang dump" of a process. I don't know of any other tool that can do anything quite like it.

I'm not trying to give an in-depth guide to the process here. Just an introduction.

The Perfmon Counter Option

You start with the performance rule. . .

You immediately have a choice between Perfmon counters or http response times.

If you've been troubleshooting a performance problem with performance monitor and know which Perfmon objects and counters to focus on already, that might be a really good way to go. If not, you'll probably want to go with http response times.

I'm going to give a quick glimpse of the Perfmon counter option first.

Sorry to be Captain Obvious here, but you select the Add Perf Triggers button.

Of course you're going to want to focus on the local computer. Don't bother trying to monitor a different server because debugdiag is only going to trip a dump on the local server.

You've got many choices here.

You could focus on high cpu. (When CPU reaches a high level, trip a memory dump.)

You could focus on memory (when virtual bytes or private bytes reaches a high level, trip a dump).

You could do something like focus on 500 (internal server error) responses being sent, as seen below.

Or you could focus on ASP.net requests queuing, for example. When the queuing begins, trip a dump. Not a bad idea. But I'd only do it if you know for a fact that asp.net requests have been queuing when the hang (or slow performance) issue occurs. This assumes prior knowledge of the symptoms.

One last observation about this Perfmon Rule is that it can be used for any process. You don't have to use it for an IIS process.

There are many options for brushing up on Perfmon to know what is normal and what is abnormal. Here are some references to consider.

http://technet.microsoft.com/en-us/library/ff758658.aspx

Monitoring and maintaining SharePoint Server 2013

http://technet.microsoft.com/en-us/library/cc730608(v=WS.10).aspx

Monitor Activity on a Web Server (IIS 7)

http://msdn.microsoft.com/en-us/library/ms972959.aspx

ASP.NET Performance Monitoring, and When to Alert Administrators

http://technet.microsoft.com/en-us/library/cc736996(v=WS.10).aspx

Monitoring ASP.NET Performance

A second way to go…

… is to stick with http response times instead of Perfmon counters.

This method only applies to websites (or web applications) running in Microsoft IIS.

We start the rule the same way. . .

But this time we're selecting the second option.

Click Add URL. . .

Now here you have a choice between ETW tracing or URL Pinging.

The ETW tracing option is new to DebugDiag 2.0. It wasn't in DD 1.2.

If you go with ETW tracing, you do not have to specify a specific page to monitor. You can leave it blank to monitor everything that is sent to IIS, you can specify a specific website on that iis server (yes, this is only useful for IIS web servers), or a specific virtual directory.

Those are some cool options!

A third way…

Or you can instruct debugdiag to send some "pings" (not actually icmp pings) against a specific web page. Here you probably need to specify a page or else just use the default page. Either way it is only going to focus its "pings" against one specific page.

One important thing to keep in mind here is that the URL to ping needs to be a url that goes directly to one web server that debugdiag is installed on. Don't specify, for example, a URL here that goes to a load balancer that then gets redirected to any of five different web servers / WFEs. If you want to use this option in a SharePoint farm, and you only have one URL that works, you might consider the possibility (use at your own risk) of creating an Alternate Access Mapping. And if you do create an AAM to go to one server, you will also probably have to add the appropriate binding in the IIS manager for the web site. (Adding an AAM wont add a binding. Extending a web site in Central Admin will add the binding but adding an AAM won't.)

You can definitely tweak the frequency of the ping and the timeout value too.

Select your application pool that you want to dump out when the url proves unresponsive.

Also you have some additional settings that you can use if you like.

I'd probably use the "generate a userdump every 45 seconds" option if I were troubleshooting a high-cpu issue. And I'd probably tell it to stop generating after three (or possibly five) userdumps.

I'd only select full userdumps, not mini userdumps. But that's just me. Others may differ. Minidumps rarely help in my experience.

↧

Footnote Numbering Trick for Word 2013

February 26, 2014, 5:36 pm

≫ Next: WebPIcmd /offline /Products:WorkflowManager versus /Products:WorkflowManagerRefresh

≪ Previous: Three Ways to Automate a “Hang Dump” in DebugDiag 2.0

Here are some steps to set footnotes to start their numbering over at 1 for each chapter of a book in Microsoft Word.

First, insert "section breaks."

The first thing that needs to happen is inserting a "section break" (not a mere page break) before the title of each different essay.

So if you have a total of seven chapters, for example, you'll need to insert seven different section breaks. (Note: Hitting CTRL+Enter for a carriage return is not the same thing.)

Move the cursor to the left of the first character of the essay title.

Move the mouse arrow to the Page Layout tab on the Word ribbon, select the drop-down triangle beside BREAKS, and select NEXT PAGE.

Insert a section break

On the Page Layout tab, in the Page Setup group, click Breaks.


Second, change the advanced footnote settings

After your section breaks are set, visit the REFERENCES tab on the Word Ribbon. Find "Footnotes" and click the arrow-like icon to its right.

[By the way, if you have any endnotes that need to be converted to footnotes, you can do that here by selecting one and clicking the CONVERT button.]
Change the value for NUMBERING from "Continuous" to "Restart each section," as seen below.
Also ensure that "Apply Changes to" is set to "Whole document," as seen below.



One possible Gottcha

If the footnotes still refuse to reset their numbering at 1, one possibility to consider is the question of whether there are some unaccepted tracking or comments that need to be accepted.   One way to investigate this possibility is to visit the Review tab in the Word ribbon, and start Accepting/Rejecting…

If you see the indicator below, tracking and commenting isn't the problem.

↧

WebPIcmd /offline /Products:WorkflowManager versus /Products:WorkflowManagerRefresh

March 26, 2014, 10:12 pm

≫ Next: Office Web Apps Exception: “An item with the same key has already been added”

≪ Previous: Footnote Numbering Trick for Word 2013

Today I had my first opportunity/need to use the webpicmd.exe tool (part of the Web Platform Installer) to attempt to download the installation bits for workflow manager, workflow client, and service bus. This needed to be done for an offline installation on servers with no access to the internet.

But when I attempted to cache the Workflow Manager locally the way that the Technet article ["Install and configure workflow for SharePoint Server 2013" @ http://technet.microsoft.com/en-us/library/jj658588(v=office.15).aspx ] said to do it (and several other blogs agreed) I ran into an unexpected problem. When I ran webpicmd.exe /offline /Products:WorkflowManager, the tool searched several feeds and said it couldn't find what I was looking for.

Error: "Loading products in offline feeds… Can't find product with id WorkflowManager."

Apparently WorkflowManager was replaced with WorkflowManagerRefresh in January 2014. That was evident when I ran Webpicmd /List /ListOption:Allto see all available packages and their abbreviations. The ones that were relevant to me were:

The following commands ran successfully to cache the installer bits locally:

Webpicmd /offline /Products:WorkflowClient /Path:D:\WebPIOfflineFiles
Webpicmd /offline /Products:WorkflowCU2 /Path:D:\WebPIOfflineFiles
Webpicmd /offline /Products:WorkflowManagerRefresh /Path:D:\WebPIOfflineFiles
Webpicmd /offline /Products:ServiceBus /Path:D:\WebPIOfflineFiles
Webpicmd /offline /Products:ServiceBus_1_1 /Path:D:\WebPIOfflineFiles
Webpicmd /offline /Products:ServiceBusCU1 /Path:D:\WebPIOfflineFiles

For a different point of reference, here is what WebPUI 4.6 showed when I searched for "workflow" or "service bus":

Additional Key Words:

Workflow Manager CU2 to get to 1.0 Refresh

Workflow Manager Client 1.0 Refresh

Service Bus 1.0

Service Bus 1.0 Cumulative Update 1

↧

Office Web Apps Exception: “An item with the same key has already been added”

April 7, 2014, 8:26 pm

≫ Next: Don’t Even Bother Trying to Enforce FIPSAlgorithmPolicy on SharePoint 2010 Servers

≪ Previous: WebPIcmd /offline /Products:WorkflowManager versus /Products:WorkflowManagerRefresh

Scenario and Symptom:

An Office Web Apps 2013 server is talking to a SharePoint 2013 server and the following shows up in the ULS log (C:\ProgramData\Microsoft\OfficeWebApps\Data\Logs\ULS) of the OWA/WAC server:

03/14/2014 10:19:20.90 w3wp.exe (0x1CDC) 0x2854 Office Web Apps Office Web Apps common agq6x Exception WAC Server FrontEnd unhandled exception [0] System.ArgumentException: An item with the same key has already been added. at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) at Microsoft.Office.Web.Apps.Environment.WacServer.WSConfigAdapter.TryGetValue[T](String settingName, T& value) at Microsoft.Office.Web.Apps.Common.WopiDocument.ValidateHostAndSetFileSource() at Microsoft.Office.Web.Apps.Common.WopiDocument..ctor(String fileRep, String accessToken, Int64 accessTokenTtl, String version, String sessionContext, WopiType wopiType) at Microsoft.Office.Web.Apps.Common.WopiDocument.IsValidWopiQueryString(String querySignature, BaseDocumentType docType, Boolean createDoc, BaseDocument& doc) at Microsoft.Office.Web.Apps.Common.WopiDocument.CreateFromQuerySignature(String querySignature) at Microsoft.Office.Web.Apps.Environment.WacServer.WSMainStorage.GetBaseDocumentFromContext(HttpContext context, String rawHostDocumentId) at Microsoft.Office.Web.Host.WacServer.WebWordViewer.WordViewerFrame.OnPreRender(EventArgs e) at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) StackTrace: at uls.native.dll: (sig=35ebf2aa-983a-4c4a-a61b-a0d5e84ac5d0|2|uls.native.pdb, offset=26E6A) at uls.native.dll: (offset=1F8A9) 29e37d01-8664-4b81-a916-bd4f92c34805

There aren't many good clues as to what key they're talking about and what item is already there.

Solution:

This problem may be caused by having duplicate entries in the allowlist for domains.

Check this ini file on the OWA/WAC server for duplicate entries: C:\ProgramData\Microsoft\OfficeWebApps\Data\FarmState\allowList.ini file

If there is a domain that is listed twice in there, that is probably the cause of this problem. But do not attempt to remove any duplicates from the .ini file.

As an alternative to opening the .ini file, you can run this PowerShell cmdlet: Get-OfficeWebAppsHost

More info:

http://technet.microsoft.com/en-us/library/jj219446(v=office.15).aspx

The Get-OfficeWebAppsHost cmdlet returns the list of host domains to which Office Web Apps Server allows file operations requests, such as file retrieval, metadata retrieval, and file changes. This list, known as the Allow List, is a security feature that prevents unwanted hosts from connecting to an Office Web Apps Server farm and using it for file operations without your knowledge.

The wildcard * is assumed for any domain that appears on the Allow List so that requests to all subdomains are also allowed. For example, if the domain contoso.com is on the Allow List, then Office Web Apps Server also allows requests to the domains corp.contoso.com and dev.contoso.com. Requests to other domains (such as fabrikam.com) are not allowed.

You can consider removing the duplicate using this cmdlet: Remove-OfficeWebAppsHost -Domain <String>

More info:

http://technet.microsoft.com/en-us/library/jj219453(v=office.15).aspx

The Remove-OfficeWebAppsHost cmdlet removes the specified host domain from the Allow List. The Allow List contains the host domains to which Office Web Apps Server allows file operations requests.

Root Cause:

In our case, we tripped an exception dump of the w3wp.exe process on the OWA server that served the Web Word Viewer while reproducing the problem.

Since psscor4.dll extension technically doesn't work for .net 4.5, and since this is technically .net 4.5 (even though it looks like 4.0), I loaded SOS.dll from C:\Windows\Microsoft.NET\Framework64\v4.0.30319 into windbg.exe while examining the memory dump.

!dae (dump all exceptions) showed these:

Exception Type : System.ArgumentException

Message: An item with the same key has already been added.

Exception Method Table: 000007FEF7EA1F08

Exception Object : 00000003FF8BE328

HResult: 80070057

InnerException: <none>

StackTrace:

000000001070DCD0 000007FEF8A5C70C System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon, System.__Canon, Boolean)

000000001070DD60 000007FE9982CAAC Microsoft.Office.Web.Apps.Environment.WacServer.WSConfigAdapter.TryGetValue[[System.__Canon, mscorlib]](System.String, System.__Canon ByRef)

000000001070DE10 000007FE99C61838 Microsoft.Office.Web.Apps.Common.WopiDocument.ValidateHostAndSetFileSource()

000000001070DEE0 000007FE99C613B0 Microsoft.Office.Web.Apps.Common.WopiDocument..ctor(System.String, System.String, Int64, System.String, System.String, Microsoft.Office.Web.Apps.Common.WopiType)

000000001070DF30 000007FE99C52A03 Microsoft.Office.Web.Apps.Common.WopiDocument.IsValidWopiQueryString(System.String, Microsoft.Office.Web.Common.EnvironmentAdapters.BaseDocumentType, Boolean, Microsoft.Office.Web.Common.EnvironmentAdapters.BaseDocument ByRef)

000000001070DFD0 000007FE99C61291 Microsoft.Office.Web.Apps.Common.WopiDocument.CreateFromQuerySignature(System.String)

000000001070E020 000007FE99C611EF Microsoft.Office.Web.Apps.Environment.WacServer.WSMainStorage.GetBaseDocumentFromContext(System.Web.HttpContext, System.String)

000000001070E060 000007FE99BF9F2A Microsoft.Office.Web.Host.WacServer.WebWordViewer.WordViewerFrame.OnPreRender(System.EventArgs)

000000001070E0E0 000007FEE2D4BCD1 System.Web.UI.Control.PreRenderRecursiveInternal()

000000001070E130 000007FEE2D6EDD9 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)

Exception Type : System.Web.HttpUnhandledException

Message:

Exception Method Table: 000007FEE2EEF828

Exception Object : 00000003FF8BE738

HResult: 80004005

InnerException: