Java List low-memory options

In this forum members can discuss topics about specific programming languages.
Post Reply
FreeT
Posts: 5
Joined: Sat Nov 19, 2011 5:48 pm

Java List low-memory options

Post by FreeT »

I'm looking to create and fill a huge array, and was wondering what would be the most efficient way to approach it in java.

Specifically I'll need to store 100 million bigDecimals, each with 100 places of accuracy. Also, I'll need to access these bigDecimals multiple times.

Trying to do this with a normal arrayList, my Eclipse program just stops running as if it's completed.
I tried increasing the available memory to 4gb, but it continues running for well over 30 minutes.
I imported vanilla.java.collections, but couldn't get it to work with bigDecimal (though I'm probably doing something wrong!).

So my question is: what options are available to store a load of bigDecimals, quickly and efficiently, and be able to access them repeatedly?

Thanks in advance.
thundre
Posts: 356
Joined: Sun Mar 27, 2011 10:01 am

Re: Java List low-memory options

Post by thundre »

FreeT wrote:I'm looking to create and fill a huge array, and was wondering what would be the most efficient way to approach it in java.

Specifically I'll need to store 100 million bigDecimals, each with 100 places of accuracy. Also, I'll need to access these bigDecimals multiple times.

Trying to do this with a normal arrayList, my Eclipse program just stops running as if it's completed.
I tried increasing the available memory to 4gb, but it continues running for well over 30 minutes.
I imported vanilla.java.collections, but couldn't get it to work with bigDecimal (though I'm probably doing something wrong!).

So my question is: what options are available to store a load of bigDecimals, quickly and efficiently, and be able to access them repeatedly?

Thanks in advance.
I'm guessing an object reference in Java for your platform takes 64 bits.

Each BigDecimal contains a long, two ints, and two references (one String which you might not use, one BigInteger which you definitely will use for 100-digit numbers). With the 8-byte object overhead that's 40 bytes plus the size of the BigInteger.

Each BigInteger contains 5 ints and an array, which will contain 11 elements for your 100-digit numbers. That's 64 bytes plus 20 overhead (the array is another object and it stores its length in an int), so 84 bytes each.

You'll need to store a reference to each of these objects, so 132 bytes per number.

If you store these in an array (BigDecimal[]), you'll need 13.2 GB.

An ArrayList is basically an object which checks array size and copies to a larger array if needed. When it expands, it uses an extra 150% for the array of references, then returns 100% when the copy is done. That's a potential 1.2 GB additional, which you can mitigate by telling the ArrayList upfront how large it needs to be. (But if you know upfront, maybe you should just use an array.)

If you don't have 13.2 GB of RAM, you could crank up the virtual memory on your computer. Or if the numbers don't change after they're computed, you might use a RandomAccessFile to store serialized versions of the BigDecimals on disk (and use a long[] to store the offset of each).
Image
Post Reply