clean up and fix logic for making mmap fail on invalid/unsupported offsets

the previous logic was assuming the kernel would give EINVAL when
passed an invalid address, but instead with MAP_FIXED it was giving
EPERM, as it considered this an attempt to map over kernel memory.
instead of trying to get the kernel to do the rigth thing, the new
code just handles the error in userspace.

I have also cleaned up the code to use a single mask to check for
invalid low bits and unsupported high bits, so it's simpler and more
clearly correct. the old code was actually wrong for sizeof(long)
smaller than sizeof(off_t) but not equal to 4; now it should be
correct for all possibilities.

for 64-bit systems, the low-bits test is new and extraneous (the
kernel should catch the error anyway when the mmap2 syscall is not
used), but it's cheap anyway. if this is an issue, the OFF_MASK
definition could be tweaked to omit the low bits when SYS_mmap2 is not
defined.

1 files changed, 7 insertions, 3 deletions

diff --git a/src/mman/mmap.c b/src/mman/mmap.c
index fd2bb07e..e99271f7 100644
--- a/src/mman/mmap.c
+++ b/src/mman/mmap.c
@@ -10,12 +10,16 @@ static void dummy0(void) { }
 weak_alias(dummy1, __vm_lock);
 weak_alias(dummy0, __vm_unlock);
 
+#define OFF_MASK ((-0x2000ULL << (8*sizeof(long)-1)) | 0xfff)
+
 void *__mmap(void *start, size_t len, int prot, int flags, int fd, off_t off)
 {
 	void *ret;
-	if (sizeof(off_t) > sizeof(long))
-		if (((long)off & 0xfff) | ((long)((unsigned long long)off>>(12 + 8*(sizeof(off_t)-sizeof(long))))))
-			start = (void *)-1;
+
+	if (off & OFF_MASK) {
+		errno = EINVAL;
+		return MAP_FAILED;
+	}
 	if (flags & MAP_FIXED) __vm_lock(-1);
 #ifdef SYS_mmap2
 	ret = (void *)syscall(SYS_mmap2, start, len, prot, flags, fd, off>>12);