Introduction
I like to run PHP under Apache via FastCGI, it feels performant enough and has the benefit of not bringing down Apache if PHP crashes (as can happen when using PHP as a module). In my job I had set this up and it was working beautifully, but little did I know what the future held…
One by one, as my servers were restarted for various reasons (Updates, changes etc…) they all threw a wobbly when starting up, saying that:
Wrapper C:/Apache2/php/php-cgi.exe cannot be accessed: (70008)Partial results are valid but processing is incomplete
Investigation
So the game was afoot… First thought was a Windows Update had broken something but this didn’t seem to be the case. A search of the Internet didn’t found many others who had hit the same problem, I would have expected more complaints if Microsoft had released an update that broken Apache / FastCGI. Another thought was the Virus scanner, but that hadn’t received any major updates apart from definition packages.
Had permissions changed? Nope. I even tried changing permissions to be totally relaxed and the same error would appear. Restricting permissions on purpose provided the expected error of being unable to access the executable. At this point the issue had spread to every work server and client I had access to, was I cursed?
I then decided to try testing the Apache set up on a fresh VM of Windows XP (no network connection, no extra software, no virus scanner, fully patched). It worked like a dream, so I start blaming the Virus scanner again but after a few more tests (adding certain executables to be ignored, disabling the scanner etc…) I felt it wasn’t to blame.
Tracking down the villain, recompiling mod_fcgid
My only hope, after fruitless searches of the Internet, appeared to be recompiling the module to find the root cause. Now I’m purely a ColdFusion, PHP, JavaScript, HTML kinda guy… The last time I had to compile something was back in the days of Turbo Pascal. Without really knowing what I was doing, I managed to stumble my way into setting up the right development environment for recompiling mod_fcgid. Here are the rough steps I took:
- Downloaded the latest version of Apache Httpd VC9 (Visual C++ 2008) from ApacheLounge. I’ve been using this version as I find it performs a bit better than the VC6 version the Apache Foundation offers and upgrading is easier when just dealing with a zip file
- I extracted this to “c:\Apache2″, as I’ll reference these files further down.
- Installed TortoiseSVN and checked out a copy of the mod_fcgid source code.
- Installed Microsoft Visual C++ 2008 Express SP1. It’s free thankfully but did take a while to download everything it needed.
- Before running anything else, right click on “My Computer” and select “Properties > Advanced > Environmental Variables” and add a new System variable called “APACHE2_HOME” set to “c:\Apache2″.
- Run Visual C++ 2008 Express
- Open a project and select the mod_fcgid.dsw file from your checkout of the source.
- It mumbled something about updating the file so I just agreed.
- Should then see a list of the files in the “Solution Explorer”.
- Right clicked on “mod_fcgid” under the “Solution Explorer” and selected “Properties”.
- In “Configuration Properties > C/C++ > General”, I added the following directories to the “Additional Include Directories” setting, keeping in mind that you may have different paths, depending on where you extracted Apache.
C:\Apache2\lib,C:\Apache2\include
Additional Include Directories
- Under “Configuration Properties > Linker > Input > Additional Dependencies” I added the following.
c:\Apache2\lib\*.lib
Additional Dependencies
- Now comes the bit where I didn’t really know what I was doing. I wanted the mod_fcgid.so created from the source but only found one way of getting it to spit it out. There’s probably another way but I couldn’t spot it.
- Click on the “Build” menu and select “Batch Build…”.
- You should be presented with a dialogue box with two items listed. Check the “Build” box for the one with “Configuration” set to “Release”.
- Click the “Build” button.
Building mod_fcgid.so
- Fingers crossed you’ll see some activity of the module being built.
- Once it’s finished, check out the folder “mod_fcgid\modules\fcgid\Release” for “mod_fcgid.so”.
Code at issue
With that all working, now was the time to hunt down that part which was throwing the error. I started with a search for the string “cannot be accessed”, which showed up in “fcgid_conf.c” under the “missing_file_msg” function. Not a bad start, so a quick search for what was calling “missing_file_msg” showed up a prime suspect under “set_wrapper_config” that looks like this:
/* Does the wrapper exist? */
if ((rv = apr_stat(&finfo, path, APR_FINFO_NORM,
cmd->temp_pool)) != APR_SUCCESS) {
return missing_file_msg(cmd->pool, "Wrapper", path, rv);
}
Commenting out the “return” line, recompiled and tested… It worked! So what the smeg was going on?
The culprit
Let me introduce you to APR_FINFO_NORM as best I can. I don’t know him that well, we’ve only just met, but after first impressions I think I’ve got a rough idea of the bloke. APR_FINFO_NORM basically asks the Apache Portable Runtime to get the full information about a file, in my case php-cgi.exe, including permissions. Shouldn’t be a problem and a very sensible thing to check, since if Apache Httpd didn’t have access to php-cgi.exe it wouldn’t be able to serve those lovely PHP files.
For the majority of users and installations, this isn’t a problem. APR_FINFO_NORM returns the information successfully (APR_SUCCESS) and Apache Httpd smiles, nods and carries on. However, if the computer is a member of Microsoft Active Directory and that AD forest is complex or large enough, you’ll hit an issue. Thanks to Microsoft, the API call that they provided called GetEffectiveRightsFromACL that “apr_stat” is using, can’t cope.
This causes “apr_stat” to return an APR_INCOMPLETE and in turn, Apache Httpd thinks it hasn’t got access and throws the (70008)Partial Results error, since it only got a partial set of permission results… I guess.
Check out the bug that has been logged for mod_fcgid and another for the “apr_stat” issue itself that’s the root cause.
Solutions
Blimey, that was a bit of sluething! Don’t worry though, as I proved above, there are ways around this until it’s resolved.
Option 1: Patch mod_fcgid
Patched code
If you look at the bug record that has been filed for mod_fcgid, you’ll see that a patch has been attached. This changes the code to avoid checking for permissions, which will obviously cause problems if Apache Httpd hasn’t got permissions, but otherwise solves our problem completely and still at least checks that the file exists. Of course this does mean you’ll have to recompile the module yourself.
/* get only require details file inode + device info */
if ((rv = apr_stat(&finfo, path, APR_FINFO_IDENT,
cmd->temp_pool)) != APR_SUCCESS) {
return missing_file_msg(cmd->pool, "Wrapper", path, rv);
}
Option 2: Change permissions on the FastCGI target
Security settings
In my case this is php-cgi.exe, but obviously there are plenty of other things that can sit behind FastCGI. Here you’ll need to make sure “apr_stat” doesn’t need to check Active Directory.
- Edit the permissions of php-cgi.exe, or whatever you’re using with FastCGI.
- Prevent it from inheriting permissions from above.
- Remove any users and groups that aren’t local to the server, so anything that is from the Active Directory.
- Also, check the local groups! You’ll might find Active Directory users / groups have been added to the local computers groups. If this is the case, they’ll need to be removed from the permissions.
- In my case, I ended up with just the local SYSTEM account and the local Administrator account. If you run Apache as another user, they’ll have to have access as well… unless they’re from Active Directory, in which case “This is not the solution you are looking for” (Waves hand).
- Start up Apache and hopefully it’ll work perfectly, error message banished.
Again, this solution is great unless you run Apache as an Active Directory user for some reason.
Option 3: Don’t let the server be a member of Active Directory
Extremely similar to the above, but taken a step further. If the server isn’t a member of the Active Directory, then none of those users or groups will have permissions. No need for “apr_stat” to go asking questions about them
Option 4: Wait until Apache fix the code
Depending on what you’re using mod_fcgid for, there are usually alternative methods available. For example, PHP can be run as an Apache module. This may not be ideal, but if any of the above aren’t an option for you, needs must.
Help promote the problems
By the looks of things, not many people have hit this issue when you search for the mysterious “(70008)Partial Results” error. But those who are affected will tend to be people in large organisations with those big complex Active Directories. Surely (I know, don’t call you Shirley) that’s the kind of large environment and organisation that you’d want Apache to have a good reputation?
If your affected, please help vote up the bugs, comment about how it has affected you or just state how you’d like to see this fixed.
- mod_fcgid bug 51020: Apache/mod_fcgid.so does not start in complex Active Directory forest
- Apache Portable Runtime bug 51560: apr_stat for APR_FINFO_NORM using GetEffectiveRightsFromAcl does not work in complex Active Directory forest
- mod_xsendfile issue 8: Also affected by the same APR bug, but during operation (unable to access files) rather than preventing start up.
Ideally the Apache Portable Runtime (APR) will be fixed, solving the problem for any other code which uses a APR_FINFO_NORM with “apr_stat”. But I’ve no idea what alternative methods they could use (something called “AuthZ APIs”?) and if that’d be available in all the Windows environments they want to support. Anyway, hopefully this will prove helpful to someone
Found something extremely useful and want to thank me with more than just words?
Recent Comments