Hacker News new | past | comments | ask | show | jobs | submit login
Why is my x64 process getting heap address above 4GB on Windows 8? (msdn.com)
54 points by ScottWRobinson on July 11, 2015 | hide | past | favorite | 34 comments



This bit me in a library of mine. Some third-party code I used was using long as memory offsets which failed on Windows 8. Unfortunately I was developing on Windows 7, so it didn't get noticed until the customer stumbled over it. :( (Fortunately said third-party code had an update that switched to ptrdiff_t in all the right places, so it was an easy fix.)

I'm a Unix person, so the notion of long being less than word-length seems pretty silly to me, but I understand why they did it, sort of.


>I'm a Unix person, so the notion of long being less than word-length seems pretty silly to me

It's silly for Windows to do something a certain way just because that's not how Unix does it?

What's silly is assuming that long and unsigned long are more than 32 bits wide, since neither C nor C++ has ever guaranteed anything more.


I don't write much C, so perhaps someone can educate me.

What are the advantages of having data types that behave differently on different platforms, like C does? It seems to make code less portable and I'm having difficulty seeing what the benefits are.

I can appreciate that, on an 8-bit processor, you'd want access to an 8-bit data type for speed - but wouldn't that be better accomplished by explicitly choosing an 8 bit data type?


Undefined behavior in C standards is often not a result of technical decisions but of practical concerns when developing a consensus during the standards process. The people on the standards committee historically included compiler vendors and users of diverse operating systems. When the process started in 1983, there were 8, 16, and 32 bit systems [and things like Burroughs 48 bit word systems].

The practical decision to leave the sizes undefined was made so that nobody was burdened unfairly. It's choices like this - not biasing stakeholders toward forgoing a standard for clear business reasons - that makes the adoption of standards likely. Subsequent C standards have adopted the same undefined behavior for the same reasons [there are still 8 and 16 bit systems developed with C] and to facilitate backward compatibility of the new standards.


Bit of a nitpick, but technically the sizes of int, long, etc are implementation defined, not undefined. Undefined behavior has different implications in C.


I agree. Undefined would mean declaring x a long integer could launch the missiles. Thanks. It's my xkcd386.


I can think in at least one reason. Having let's say `int` as long as the register size would get you portable efficiency since it doesn't matter if the underlying processor is 8, 16 , 32 or 64 bit, you always will have a type that fix just right.

There are some other types (i.e int32_t, int64_t, etc) that you could use if you want to be sure that your variable would have the same size regardless of the architecture.


For that, there's now intN_fast_t in stdint.h - the fastest data type that's at least N bits.


>It seems to make code less portable

Than what? Than a freshly designed C-like language? Yes. But ISO C wasn't an effort to invent a new language. It was an effort to standardize existing practice, and provide a platform for improving C. One of the core goals of ISO C is to maintain every last bit of backward compatibility possible, which is why integer widths are not precisely specified: had they been, multiple vendors' implementations and the code that was written for them would have broken.

But to your main point, I agree: explicitly sized integer types are almost always what you should use. For example, don't store a pointer in an unsigned long (and certainly not a signed long); store it in a uintptr_t, or hell, just store it as a pointer. In fact, the fundamental types like int, long, etc. are effectively just shorthand for int_fast16_t, int_fast32_t, etc., but without the mnemonic value.


stdint.h


If it's not specified, implementers have the option of making it wider. Matching the width of a pointer seems like a better choice.


Yeah however the C standard recommends that the integer conversion rank of size_t and ptrdiff_t is lower or equal to long.

http://port70.net/~nsz/c/c11/n1570.html#7.19p4


From a certain angle it would seem size_t and ptrdiff_t are meant to roughly correspond to array subscript use cases. Therefore it's not unheard of for them to not be as wide as a pointer. intptr_t is for that.

In your link they have examples where size_t and ptrdiff_t are 16 bits. Obviously this is absurd on today's machines. But you can imagine a machine where array subscript is most convenient at 16 bits and total addressable space is larger (16 bit x86 comes to mind). By the same token I don't think it would be too weird to have a 32 bit size_t on a 64 bit architecture. It would just make memcpy et al more annoying when your allocations are large.

Anyway the quotation from your link:

> The types used for size_t and ptrdiff_t should not have an integer conversion rank greater than that of signed long int unless the implementation supports objects large enough to make this necessary.

I guess if longs are 32bit and you can have a 4g allocation this pretty well follows the spirit here.


Pointer packing is popular with people wanting to write efficient interpreters for programming languages. The memory use and performance gains are massive and it was perfectly safe during the 32-bit era.

LuaJIT does it for example. Interestingly enough some programmers do this without even understanding how memory allocation works. For example the author of the newish wren language: https://github.com/munificent/wren

Somehow most programmers - including almost everyone who frequents reddit.com/r/programming - believe that "virtual address space" means that the addresses returned by malloc() will restart from (somewhere close to) 0x0 for every new process i.e. that it is safe to truncate pointers if you never intent to allocate more than ~2GB per process anyway.


LuaJIT does something weirder. It does double packing - using 48 bits (IIRC) of value inside a "NaN" double encoding. It is going to be a while before this becomes a problem, even on 64-bit systems - you need more than 128TB addressable space before this becomes a problem (and I suspect Pall will switch to a 51-bit (1024TB=1PB) pointer when that future is on the horizon).


No that is wrong. Currently the x64 port of LuaJIT is limited to about 1GB [1] in the best case and can fail to allocate any Lua objects at all if the lower region of the address space is already completely consumed.

[1] http://lua-users.org/lists/lua-l/2010-11/msg00241.html


Thanks. A slightly later discussion I found is here[0]. However, this is not a limit imposed by the tag structure used by LuaJIT, but rather the Lua/LuaJIT GC (from a quick reading of the mailing list discussion - I might be wrong here). Apparently, there's a GC in the works that is supposed to resolve this, but it isn't there yet.

[0] http://lua-users.org/lists/lua-l/2012-04/msg00729.html


> Why are they truncating 64-bit pointers to 32-bit values?

Maybe to get more compact data structures? But in that case they shouldn't rely on that and try detecting that behavior at runtime.

E.g. the hotspot jvm tries to map memory in the low end of address space to compress pointers but falls back to full pointers when that fails.


As mentioned on the blog post's comments, it's often caused either by storing a pointer in a DWORD variable (very common on 32-bit Windows), or by storing a pointer in a "long" variable (on most common 32-bit and 64-bit platforms, a "long" is at least as wide as a pointer; the main exception is 64-bit Windows).

The correct variable type to store a pointer would be intptr_t/uintptr_t for portable software (and DWORD_PTR for Windows-only software), but the "intptr" types didn't exist until the 1999 C standard (and, as far as I could find, the default Windows compiler didn't add the corresponding header file until around 2010). Software older than that would often assume that a "long" (or a DWORD, which IIRC was just another name for "long") was big enough to store a pointer, or perhaps tried to use the C99 types and used "long" if the header wasn't found.


> storing a pointer in a DWORD variable (very common on 32-bit Windows)

Is it? I mean, if you're lucky it won't cause a compiler warning on x86, but the sane thing to do is to use the types which the API docs specify. Which would usually be things like LPVOID or HANDLE.

That's basically how I managed to port a service and several drivers to 64-bit Windows many years ago with almost no changes.


At the time the code was written the warnings you get now didn't exist. Then the 32-bit code worked, so nobody needed to change the it. Now with windows 8 they see it for the first time that it doesn't work.

I know as I also worked on the huge codebase where the oldest pieces of the code were written around 1995.


> Maybe to get more compact data structures?

Even if this is why, there is never a valid excuse for blindly truncating 64-bit pointer to 32-bits. It's just like saying "I'm gonna just go ahead and truncate all bank account balances to 16-bit integers to save memory in my data structures." It may be the reason, but that reason is still incredibly stupid.


> It may be the reason, but that reason is > still incredibly stupid.

Sometimes there are stupid reasons, but sometimes even reasons which are incredibly stupid from a technical standpoint make sense in a broader context.

For a short-term fix, for example, I can very well imagine to prefer this "trick" forcing sub-4G allocations if the other alternative would be to change 1000 places in undocumented legacy code doing crazy casts... And if it's for a product (or product component) on life-support only needed for a forseeable future, to me it makes perfect sense to ask the initial "crazy" question, even if it makes me cringe...


> For a short-term fix, for example, I can very well imagine to prefer this "trick" forcing sub-4G allocations ...

But that's not what this is doing - it's taking an address allocated at an arbitrary place and then pretends that it was allocated sub-4G, whether or not it actually was. If it happens that it wasn't, this will likely cause memory corruption or an access violation.

Even if your product is nearing end-of-life, I'd still consider such a move a big "f* you" to your customers as you'd basically be tolerating that your product may crash without warning at any time.

From what I understood, what makes this idea so bad isn't truncating to 32bit - it's truncating to 32 bit while setting the LARGEADDRESSAWARE build flag - which basically tells the OS "I'm fine with all kinds of 64bit addresses, bring them on!"

The correct fix is outlined at the end of the article - and it does not involve combing through undocumented legacy code:

> If there is some fundamental reason that they have to truncate pointers to 32-bit values, then they should build without /LARGEADDRESSAWARE so that the process will be given an address space of only 2GB, and then they can truncate their pointers all they want.


I'd suggest that even if it's absolutely necessary, one could at least put something like

    #if (ULONG_MAX < UINTPTR_MAX) && !defined (SUPPRESS_TRUNCATION_ERROR)
    #error Truncating pointers to long is a bad idea here
    #endif
in some source file or another, just so you have some indication whether or not things are going to break when you change the build environment.


  static_assert(sizeof(DWORD) == sizeof(void*), "assume 32-bit pointers");


If we're talking about "undocumented legacy code doing crazy casts," static_assert is probably not an option.


Perhaps the maintainer of the legacy application can override operator new to use a custom memory allocator that uses a block of memory allocated below 4GB using VirtualAlloc.


Hotspot is really quite subtle, and thorough in its strategy. Because of the alignment of java objects it can both shift and truncate pointers (reversing the shift when they are actually used), but can also handle the heap not being allocated in the low end of address space by storing the offset of the base address, and adding/subtracting that as needed as well.


There are tons of C/C++ codes that use `int` for every type of integer, including indexing arrays and storing pointer values. A relic of the time where everything was 32-bit. It is common on Unix system as well. When you compile C/C++ library, a Python c-extension, there are usually many warnings about integer truncation.


aka bugs!


This is one reason why OS X enforces a hard page-zero on 64-bit programs - it is a hard error to map or allocate anything in the lowest 4 GB of memory, so programmers will immediately discover pointer truncation bugs when porting 32-bit code to 64-bit.


What if you have a load command in your mach-o which asks for an address below the 32-bit boundary? Will loading the image fail?


Long was not 64 bits? This is why C standard has stdint.h which typedefs numeric types of specific length (for 64 bit integers use int64_t, etc). It seems Visual Studio also has it nowadays https://msdn.microsoft.com/en-us/library/hh874765.aspx.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: