Some disjointed thoughts on large pages.
My undertsanding is than enabling large pages in the OS reserves a block of memory for use
reducing the amount available for applications, presumably ones that don't use large pages.
This leads to an engineering issue. How many large pages should be allocated in the OS?
How many large pages would cpuminer use? Is it different based on the algo. What if there aren't
enough large pages for cpuminer?
I found an easier way to enable large pages in Linux, no grub required. It's old and for RH but the commands,
kernel variables and files exist on Ubuntu so it should work.
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/sect-Oracle_9i_and_10g_Tuning_Guide-Large_Memory_Optimization_Big_Pages_and_Huge_Pages-Configuring_Huge_Pages_in_Red_Hat_Enterprise_Linux_4_or_5.html
The Wikipedia article says large pages aren't widely used and used mostly in server environments but it
doesn't say why. I suspect it doesn't perform well in a desktop environment with multiple applications
running.
The arcticle also mentioned elevated priviledges, though it's not clear they mean the application must run
as root or only the OS config. It also mentioned issues swapping large pages to disk
The arcticle also mentions the application needs to set a flag, no such flag exists in malloc so it's not so
simple.
I have some thoughts on possible implementation. This looks like a specialist feature that is not applicable
to some environments therefore cpuminer would have to disable it by default. cpuminer would also have
to confirm large pages is enabled in the OS before using them.
Edit: I'm reluctant to advertise such a feature because then I'd be responsible fo helping users setup their OS.
Do large pages apply to static application data or only dynamically allocated memory?
My undertsanding is than enabling large pages in the OS reserves a block of memory for use
reducing the amount available for applications, presumably ones that don't use large pages.
This leads to an engineering issue. How many large pages should be allocated in the OS?
How many large pages would cpuminer use? Is it different based on the algo. What if there aren't
enough large pages for cpuminer?
I found an easier way to enable large pages in Linux, no grub required. It's old and for RH but the commands,
kernel variables and files exist on Ubuntu so it should work.
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/sect-Oracle_9i_and_10g_Tuning_Guide-Large_Memory_Optimization_Big_Pages_and_Huge_Pages-Configuring_Huge_Pages_in_Red_Hat_Enterprise_Linux_4_or_5.html
The Wikipedia article says large pages aren't widely used and used mostly in server environments but it
doesn't say why. I suspect it doesn't perform well in a desktop environment with multiple applications
running.
The arcticle also mentioned elevated priviledges, though it's not clear they mean the application must run
as root or only the OS config. It also mentioned issues swapping large pages to disk
The arcticle also mentions the application needs to set a flag, no such flag exists in malloc so it's not so
simple.
I have some thoughts on possible implementation. This looks like a specialist feature that is not applicable
to some environments therefore cpuminer would have to disable it by default. cpuminer would also have
to confirm large pages is enabled in the OS before using them.
Edit: I'm reluctant to advertise such a feature because then I'd be responsible fo helping users setup their OS.
Do large pages apply to static application data or only dynamically allocated memory?
ah yes i forgot, once enabled you can also set the amount dynamically in the os with the commands referenced
i fully understand and support your statement about advertising this feature, a simple cli option to enable it would be enough in my opinion (just link the os setup guides in the readme or somewhere, everything else is up to the user)
well its not widely used because for most applications there is no need to use it, only memory heavy applications can benefit from it, which largely limits the usecases for most applications (except maybe chrome
)i dont think it reduces performance, the ram is just gone, so if your system has lets say 12gb and your application uses 1gb, the remaining 11gb should still work as usual
about swapping i dont know, i never had to (and it would ruin performance as well), better know your specs and use hugepages accordingly
last note: you can enable transparent hugepages in linux (not sure if enabled by default), you can read about that here: https://www.kernel.org/doc/Documentation/vm/transhuge.txt
seems you also need to adjust the allocations
looks like you mixed something up now no probs your head is full from solving poor peoples problems lol
