How much precomputation are you willing to do? Is it OK to use a computer for this? Note that you can verify this multiple ways (by patching some quick-and-dirty code into libsecp, and using sage are my go-tos) and that the precomputation will have nothing to do with your secret key at all, so you can also ask outside parties to verify.
Depending on your answer, this might not be too crazy, and you could do it in less than a day per pubkey. Here's one scheme:
0. As others have mentioned, the public key is derived from the secret key by means of x → x*G, where x, your secret key, is a 256-bit number, G is a standard generator of the secp256k1 elliptic curve, and * means repeated addition of G to itself by the elliptic curve group addition formula.
0a. The formula can be found on Wikipedia and
here. I recommend you understand the wikipedia formula algebraically, then go to that link to find more computationally efficient schemes. In particular you'll want to use a coordinate system that lets you avoid doing any modular inversion, because that's a bear to do by hand. (You will eventually need to do this, because the final form needs to be in affine coordinates, but you can put it off to the end instead of doing it at every step.)
0b. The formulae that libsecp256k1 uses are
in this Sage script, which if you run, will also check that they are actually algebraically equivalent to the cleaner formula found on wiki and elsewhere. libsecp's computational goals (avoid inversions, strongly prefer addition to multiplication) are similar enough to a human's goals that it's probably faster to just use these than to go researching better ones.
0c. The formulae for doubling and adding distinct points are different. You'll need to double to do the precomputation, but never afterward. So that will simplify your life a bit.
Ok, the actual scheme:
1. Take your secp256k1 generator G, and compute G, 2G, 3G, ..., 15G. Then compute each of these times 16, times 16^2, times ..., times 16^63. (This isn't as bad as it looks since each generation can be gotten from the last by multiplying by 16 again.) To multiply by 16, double 4 times. This is a total of 15×64 = 960 points you need to precompute. Now, I'm not in your head, but I really think your ghosts will allow you to do this using computers, since it's independent of your secret and you can do it multiple ways.
1a. I think it would be funny for the libsecp project to sell books with these precomputed tables, like the books of logarithms that sailors used to use. It's a quick job in sage and LaTeX, if I still think it's funny by the end of this post I'll make a PDF.
2. Now, take your secret key in hex-encoding. This is a base 16 encoding, something like x = a + b*16 + c*16^2 + ... and so on, where all of a, b, c, are between 0 and 15 inclusive. There will be 64 digits, some of which (four of them, on average) are zero. So you can write x*G = (a + b*16 + c*16^2 + ...)*G = a*G + b*16*G + c*16^2*G.
2a. Here's the cool part: the 0*G's you can ignore. Everything else that's multiplied by G in the above formula is in your precomputed table. So you'll have (at most) 64 points that you simply look up, and you have to add them together. 63 additions. To do this in 24 hours, you'd have to do each one in a bit under 23 minutes. This is plausible, but I couldn't do it.
By increasing the size of your precomputed table you can reduce the work from 64 points, but the table gets much bigger. (To halve your work, you roughly have to square the size of the table.) To add 32 points, you'd need 8160 precomputed points; for 16 points you'd need a bit over a million; for 8 it's 35 billion ... for 1 you'd need to write out literally every single point.)
Edit: Here, I have done the precomputation for you. You want page 38, base 16 digits.
https://download.wpsoftware.net/bitcoin/todl.pdf 
.
