overhaul cancellation to fix resource leaks and dangerous behavior with signals

this commit addresses two issues:

1. a race condition, whereby a cancellation request occurring after a
syscall returned from kernelspace but before the subsequent
CANCELPT_END would cause cancellable resource-allocating syscalls
(like open) to leak resources.

2. signal handlers invoked while the thread was blocked at a
cancellation point behaved as if asynchronous cancellation mode wer in
effect, resulting in potentially dangerous state corruption if a
cancellation request occurs.

the glibc/nptl implementation of threads shares both of these issues.

with this commit, both are fixed. however, cancellation points
encountered in a signal handler will not be acted upon if the signal
was received while the thread was already at a cancellation point.
they will of course be acted upon after the signal handler returns, so
in real-world usage where signal handlers quickly return, it should
not be a problem. it's possible to solve this problem too by having
sigaction() wrap all signal handlers with a function that uses a
pthread_cleanup handler to catch cancellation, patch up the saved
context, and return into the cancellable function that will catch and
act upon the cancellation. however that would be a lot of complexity
for minimal if any benefit...

2 files changed, 6 insertions, 0 deletions

diff --git a/arch/i386/pthread_arch.h b/arch/i386/pthread_arch.h
index 155b4847..64d75cbc 100644
--- a/arch/i386/pthread_arch.h
+++ b/arch/i386/pthread_arch.h
@@ -4,3 +4,6 @@ static inline struct pthread *__pthread_self()
 	__asm__ ("movl %%gs:0,%0" : "=r" (self) );
 	return self;
 }
+
+#define PC_AT_SYS(c) \
+	(*(uint16_t *)(((ucontext_t *)(c))->uc_mcontext.__gregs[14])==0x80cd)
diff --git a/arch/x86_64/pthread_arch.h b/arch/x86_64/pthread_arch.h
index 40d26797..af7ae86e 100644
--- a/arch/x86_64/pthread_arch.h
+++ b/arch/x86_64/pthread_arch.h
@@ -4,3 +4,6 @@ static inline struct pthread *__pthread_self()
 	__asm__ ("movq %%fs:0,%0" : "=r" (self) );
 	return self;
 }
+
+#define PC_AT_SYS(c) \
+	(*(uint16_t *)(((ucontext_t *)(c))->uc_mcontext.__gregs[16])==0x050f)