January 17, 2013
I just confirmed the cause an extremely weird problem that’s been frustrating me for days. I want to share it so nobody else has to waste their time on this.
It appears that – at least on my build machine, a Windows 7 SP1 x64 box with Windows SDK 7.1, Visual Studio 2010 Express SP1 and Visual Studio Express 2012 on it – vcregress check will hang indefinitely with a postgres.exe process sitting at 100% cpu if the regression tests are run in a path that is too deep. This seems to happen with both x64 and x86 builds.
git.exe seems to have a similar problem, where a git clean -fdx in a deep directory tree will sit at 100% cpu forever, making no progress.
In both git.exe‘s and postgres.exe's cases, Process Monitor shows a steam of QueryNameInformationFile events with result BUFFER OVERFLOW. QueryNameInformationFile is the IRP_MJ_QUERY_INFORMATION operation of ZwQueryInformationFile as documented in MSDN here. It’s a kernel-level operation.
I’m yet to determine the root cause of the issue. To work around the problem, build in a shallower directory tree.
I’ve included a bunch of details after the cut, primarily to help anyone else with this problem find this post.
This works:
cd C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\BT\release\SL_OS\windows\TA\x86\TO\2008\src\tools\msvc
"perl" vcregress.pl check
============== creating temporary installation ==============
============== initializing database system ==============
============== starting postmaster ==============
running on port 57532 with PID 100
============== creating database "regression" ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries ==============
test tablespace ... ok
This, with the exact same code, fails, hanging indefinitely on test tablespace:
cd C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\tools\msvc
"perl" vcregress.pl check
============== creating temporary installation ==============
============== initializing database system ==============
============== starting postmaster ==============
running on port 57532 with PID 128
============== creating database "regression" ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries ==============
test tablespace ...
The postgres.exe backend the test is running on sits at 100% cpu (or rather, 50% because it’s 100% of one core on a dual core box). It cannot be terminated via Task Manager or Process Explorer – End Task appears to succeed without error, but doesn’t actually kill the process. Neither does taskkill.exe even with the /F flag. Only a reboot seems to get rid of it. Sometimes subsequent attempts to kill it fail with a “permission denied” error.
When they’re hung, a backtrace from Process Explorer shows, for a 32-bit Pg running on a 64-bit host:
ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a
ntdll.dll!NtCreateFile+0xa
wow64.dll!Wow64EmulateAtlThunk+0xe697
wow64.dll!Wow64SystemServiceEx+0xd7
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x2d
wow64.dll!Wow64SystemServiceEx+0x1ce
wow64.dll!Wow64LdrpInitialize+0x429
ntdll.dll!RtlUniform+0x6e6
ntdll.dll!RtlCreateTagHeap+0xa7
ntdll.dll!LdrInitializeThunk+0xe
ntdll.dll!NtCreateFile+0x12
kernel32.dll!CreateFileW+0x4a
kernel32.dll!CreateFileA+0x36
postgres.exe!pgwin32_open+0xbc
postgres.exe!BasicOpenFile+0x1e
postgres.exe!GetNewRelFileNode+0xfa
postgres.exe!heap_create_with_catalog+0x1df
postgres.exe!DefineRelation+0x44e
postgres.exe!standard_ProcessUtility+0x425
postgres.exe!PortalRunUtility+0xa2
postgres.exe!PortalRunMulti+0x11b
postgres.exe!PortalRun+0x176
postgres.exe!exec_simple_query+0x3d1
postgres.exe!PostgresMain+0x5b5
postgres.exe!BackendRun+0x179
postgres.exe!SubPostmasterMain+0x31d
postgres.exe!main+0x1f4
postgres.exe!__tmainCRTStartup+0x122
kernel32.dll!BaseThreadInitThunk+0x12
ntdll.dll!RtlInitializeExceptionChain+0x63
ntdll.dll!RtlInitializeExceptionChain+0x36
In Process Monitor, you see a long stream of QueryNameInformationFile events:
Date & Time: 16/01/2013 9:31:17 PM
Event Class: File System
Operation: QueryNameInformationFile
Result: BUFFER OVERFLOW
Path: C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\data
TID: 4908
Duration: 0.0000015
Name: \jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\da
with stack traces like this, again from a 32-bit Pg on a 64-bit host:
0 0xfffff88001420067 0xfffff88001420067
1 0xfffff88001421329 0xfffff88001421329
2 0xfffff8800141f6c7 0xfffff8800141f6c7
3 0xfffff80002589636 0xfffff80002589636
4 0xfffff800026b2bbe 0xfffff800026b2bbe
5 0xfffff800026b2f2f 0xfffff800026b2f2f
6 0xfffff800026b3947 0xfffff800026b3947
7 0xfffff800026b3a4b 0xfffff800026b3a4b
8 0xfffff800025fc3ef 0xfffff800025fc3ef
9 0xfffff800025d8162 0xfffff800025d8162
10 0xfffff800025d95f6 0xfffff800025d95f6
11 0xfffff800025daefc 0xfffff800025daefc
12 0xfffff800025e5b54 0xfffff800025e5b54
13 0xfffff800022e1253 0xfffff800022e1253
14 ntdll.dll ZwCreateFile + 0xa 0x773c186a C:\Windows\SYSTEM32\ntdll.dll
15 wow64.dll whNtCreateFile + 0x10f 0x7388bff7 C:\Windows\SYSTEM32\wow64.dll
16 wow64.dll Wow64SystemServiceEx + 0xd7 0x7387cf87 C:\Windows\SYSTEM32\wow64.dll
17 wow64cpu.dll TurboDispatchJumpAddressEnd + 0x2d 0x73802776 C:\Windows\SYSTEM32\wow64cpu.dll
18 wow64.dll RunCpuSimulation + 0xa 0x7387d07e C:\Windows\SYSTEM32\wow64.dll
19 wow64.dll Wow64LdrpInitialize + 0x429 0x7387c549 C:\Windows\SYSTEM32\wow64.dll
20 ntdll.dll LdrpInitializeProcess + 0x17e4 0x773b4956 C:\Windows\SYSTEM32\ntdll.dll
21 ntdll.dll ?? ::FNODOBFM::`string' + 0x29220 0x773b1a17 C:\Windows\SYSTEM32\ntdll.dll
22 ntdll.dll LdrInitializeThunk + 0xe 0x7739c32e C:\Windows\SYSTEM32\ntdll.dll
23 ntdll.dll ZwCreateFile + 0x12 0x775700a6 C:\Windows\SysWOW64\ntdll.dll
24 KERNELBASE.dll CreateFileW + 0x35e 0x7628c5ef C:\Windows\syswow64\KERNELBASE.dll
25 kernel32.dll CreateFileWImplementation + 0x69 0x76a83f86 C:\Windows\syswow64\kernel32.dll
26 kernel32.dll CreateFileA + 0x37 0x76a853e4 C:\Windows\syswow64\kernel32.dll
27 postgres.exe pgwin32_open + 0xbc, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp
\src\port\open.c(77) 0x13eba5c C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src
\test\regress\tmp_check\install\bin\postgres.exe
28 postgres.exe BasicOpenFile + 0x1e, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\x
p\src\backend\storage\file\fd.c(560) 0x12eec6e C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_
TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
29 postgres.exe GetNewRelFileNode + 0xfa, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_
os\xp\src\backend\catalog\catalog.c(578) 0x11a836a C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_
TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
30 postgres.exe heap_create_with_catalog + 0x1df, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg
_target_os\xp\src\backend\catalog\heap.c(1073) 0x11adf8f C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH
\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
31 postgres.exe DefineRelation + 0x44e, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\commands\tablecmds.c(636) 0x11fb98e C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
32 postgres.exe standard_ProcessUtility + 0x425, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\utility.c(535) 0x130f425 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
33 postgres.exe PortalRunUtility + 0xa2, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(1191) 0x130d022 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
34 postgres.exe PortalRunMulti + 0x11b, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(1320) 0x130d16b C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
35 postgres.exe PortalRun + 0x176, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(815) 0x130da06 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
36 postgres.exe exec_simple_query + 0x3d1, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\postgres.c(1053) 0x130ba61 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
37 postgres.exe PostgresMain + 0x5b5, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\postgres.c(3969) 0x130c345 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
38 postgres.exe BackendRun + 0x179, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\postmaster\postmaster.c(3987) 0x12c9459 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
39 postgres.exe SubPostmasterMain + 0x31d, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\postmaster\postmaster.c(4491) 0x12cd08d C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
40 postgres.exe main + 0x1f4, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\main\main.c(175) 0x123fc24 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
41 postgres.exe __tmainCRTStartup + 0x122, f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c(554) 0x13f7366 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
42 kernel32.dll BaseThreadInitThunk + 0xe 0x76a833aa C:\Windows\syswow64\kernel32.dll
43 ntdll.dll __RtlUserThreadStart + 0x70 0x77589ef2 C:\Windows\SysWOW64\ntdll.dll
44 ntdll.dll _RtlUserThreadStart + 0x1b 0x77589ec5 C:\Windows\SysWOW64\ntdll.dll