Java - writing int arrays

In this forum members can discuss topics about specific programming languages.
Post Reply
User avatar
PurpleBlu3s
Posts: 73
Joined: Mon Sep 19, 2011 5:49 pm

Java - writing int arrays

Post by PurpleBlu3s » Fri Nov 11, 2011 6:03 pm

Hi, I'm hoping to write two fairly large integer arrays to a file (about 1000000-2000000 ints per array) to a file, and read them back again. I don't have much file IO experience in general other than reading simple text files, so I would appreciate it if anyone can help me with this.

Thanks.

EDIT: I'll just add that I've tried a naive approach of building a string of the array, but it takes longer than I can be bothered to wait to build the string with 1000000 ints, unsurprisingly.
Image

User avatar
Argaen
Posts: 10
Joined: Fri Apr 10, 2009 10:19 pm

Re: Java - writing int arrays

Post by Argaen » Mon Nov 14, 2011 4:48 pm

Assuming you already have populated the arrays, you can write them to a File with a code like this:

Code: Select all

void writeToFile(int[] array, String fileName) throws IOException{
   FileWriter fw = new FileWriter(new File(fileName));
   for(int i in array){
      fw.write(i + " ");
   }
   fw.close(); // Really important!
}
That would write every number in a single line, separated by a single space between them. The write() method is like System.out.print(), it does not print a line feed.
To append something to the end of the file, without losing what was already there you can call the FileWriter constructor with a second parameter like this:

Code: Select all

FileWriter fw = new FileWriter(new File(fileName), true);
The official documentation of the FileWriter class is here: http://download.oracle.com/javase/6/doc ... riter.html.

To read them I would recomend the Scanner class, with something like this:

Code: Select all

ArrayList<Integer> readFile(String fileName) throws IOException{
   Scanner s = new Scanner(new File(fileName));
   ArrayList<Integer> list = new ArrayList<Integer>();
   while(s.hasNextInt()){
      list.add(s.nextInt());
   }
   return list;
}
ImageEst Sularis Oth Mithas.

User avatar
PurpleBlu3s
Posts: 73
Joined: Mon Sep 19, 2011 5:49 pm

Re: Java - writing int arrays

Post by PurpleBlu3s » Tue Nov 15, 2011 3:04 pm

Argaen wrote: Helpful stuff.
Thanks. I was essentially doing what you did in your writeToFile method, but I was using BufferedWriter and I was building the string and then writing the string. I don't understand why what I did before would be so much slower than this - it took about 1 second to write a 1.6 million length array to a file this way, but it took longer than I could be bothered to wait the old way (>20 minutes).
Image

thundre
Posts: 356
Joined: Sun Mar 27, 2011 9:01 am

Re: Java - writing int arrays

Post by thundre » Tue Nov 15, 2011 5:15 pm

PurpleBlu3s wrote:Thanks. I was essentially doing what you did in your writeToFile method, but I was using BufferedWriter and I was building the string and then writing the string. I don't understand why what I did before would be so much slower than this - it took about 1 second to write a 1.6 million length array to a file this way, but it took longer than I could be bothered to wait the old way (>20 minutes).
Building a String with a StringBuilder, or with String concatenation in a loop? String construction by repeated concatenation in Java is notoriously slow, which is why StringBuffer and StringBuilder were added to the API.
Image

User avatar
hk
Administrator
Posts: 10488
Joined: Sun Mar 26, 2006 9:34 am
Location: Haren, Netherlands

Re: Java - writing int arrays

Post by hk » Tue Nov 15, 2011 8:18 pm

I'm no Java programmer, but converting integers to string to write them to a file did not seem a very good idea to me and I wondered if there was a way to write them as integers.
Some Googling brought me to this:
http://www.java-examples.com/write-int- ... tputstream
and its companion:
http://www.java-examples.com/read-int-f ... nputstream
Image

TripleM
Posts: 382
Joined: Fri Sep 12, 2008 2:31 am

Re: Java - writing int arrays

Post by TripleM » Tue Nov 15, 2011 10:39 pm

- ignore this -

A DataOutputStream won't actually make the file you are writing to readable as text if that is important; they'd just be written as bytes. It is also about 50 times slower (based on a quick test) than a plain FileWriter.

You could always wrap the FileWriter in a PrintWriter, which provides methods like print(int x) - though all that does is convert x to a String and call write(), since a file like that is, after all, just one long string.

The method mentioned above is the best to use, really.
Last edited by TripleM on Wed Nov 16, 2011 9:32 am, edited 2 times in total.

User avatar
hk
Administrator
Posts: 10488
Joined: Sun Mar 26, 2006 9:34 am
Location: Haren, Netherlands

Re: Java - writing int arrays

Post by hk » Wed Nov 16, 2011 9:02 am

Odd language, that Java.
I would say that writing 4 bytes would be much faster than the equivalent string that might have 9 bytes.
Not to speak about converting to string and back again.
So there must be something wrong with Java's implementation.
Image

TripleM
Posts: 382
Joined: Fri Sep 12, 2008 2:31 am

Re: Java - writing int arrays

Post by TripleM » Wed Nov 16, 2011 9:11 am

Actually sorry, ignore the figure above, it appears the FileWriter was buffered while the DataOutputStream wasn't. If I buffer the latter, it is of course faster.

So just depends on whether you want the file readable as text or not.

User avatar
hk
Administrator
Posts: 10488
Joined: Sun Mar 26, 2006 9:34 am
Location: Haren, Netherlands

Re: Java - writing int arrays

Post by hk » Wed Nov 16, 2011 2:41 pm

Just another thing:
Before writing the integers to a file it might be useful to inspect the data.
I encountered this when I wanted to write a large number of consecutive primes to a file, to read them in again when necessary.
As it turned out no prime gap in the range exceeded 510. So I divided every gap by 2 and stored only the prime gaps divided by 2 as bytes (buffered reading and writing of course).
The file then contained the following numbers as bytes: 2,3,1,1,2,1,2... meaning 2,3,3+2*i=5,5+2*1=7,7+2*2=11,11+2*1=13,13+2*2=17,....
It turned out to be the fastest way to save and retrieve enough information about those primes to recalculate them when reading them in again.
It turned out too, that recalculating them using an efficient prime sieve was even faster. So I haven't ever used that file after its creation. It's still somewhere on my computer.
Image

User avatar
PurpleBlu3s
Posts: 73
Joined: Mon Sep 19, 2011 5:49 pm

Re: Java - writing int arrays

Post by PurpleBlu3s » Thu Nov 17, 2011 12:54 am

Thanks for the replies. Yes I was using concatenation in a loop. It's all irrelevant now because the reason I was doing it was to make it easier to try out different colouring methods on Nebulabrots, but it turns out not to work as I only realised after coding it. If anyone happens to know much about Nebulabrot and if it's feasible to save the data that describes it so that different colouring methods can be used without having to redraw the whole thing, then I would love to know. Nebulabrot are Buddhabrot but with different colouring. Buddhabrot are an interesting alternative way of drawing the Mandelbrot set. Here is an example of a Nebulabrot my program has produced (2000x2000 pixels; took about 16 hours to produce - this could definitely be a lot faster, but it's just the best way I've been able to code):
http://dl.dropbox.com/u/16319566/8000it ... ts-01c.png
Image

thundre
Posts: 356
Joined: Sun Mar 27, 2011 9:01 am

Re: Java - writing int arrays

Post by thundre » Thu Nov 17, 2011 1:43 am

hk wrote:It turned out to be the fastest way to save and retrieve enough information about those primes to recalculate them when reading them in again.
It turned out too, that recalculating them using an efficient prime sieve was even faster. So I haven't ever used that file after its creation. It's still somewhere on my computer.
I had a similar experience with XML data. Saving a very large XML stream, I wondered if compressing the data using ZipOutputStream was feasible. Testing showed that not only was it feasible, it was faster, because the process was I/O bound, and the compressed data required less I/O.

Java has a good RandomAccessFile class. If you wanted to store a prime list that actually exceeds your available memory, you could use that.
Image

Post Reply