Ytukyg

What is the most efficient (or recommended) way to prepend a string or a file to another large file in Scala, preferably without using external libraries? The large file can be binary.

E.g.

if prepend string is:
header_information|123.45|xyzn

and large file is:

abcdefghijklmnopqrstuvwxyz0123456789

abcdefghijklmnopqrstuvwxyz0123456789

abcdefghijklmnopqrstuvwxyz0123456789

...

I would expect to get:

header_information|123.45|xyz

abcdefghijklmnopqrstuvwxyz0123456789

abcdefghijklmnopqrstuvwxyz0123456789

abcdefghijklmnopqrstuvwxyz0123456789

...

I come up with the following solution:

1. Turn prepend string/file into InputStream


2. Turn large file into InputStream


3. "Combine" InputStreams together using java.io.SequenceInputStream
   


4. 
   

   Use java.nio.file.Files.copy to write to target file

   
   
   

   object FileAppender {
   
     def main(args: Array[String]): Unit = {
   
       val stringToPrepend = new ByteArrayInputStream("header_information|123.45|xyzn".getBytes)
   
       val largeFile = new FileInputStream("big_file.dat")
   
       Files.copy(
   
         new SequenceInputStream(stringToPrepend, largeFile),
   
         Paths.get("output_file.dat"),
   
         StandardCopyOption.REPLACE_EXISTING
   
       )
   
     }
   
   }
   
   

   
   

Tested on ~30GB file, took ~40 seconds on MacBookPro (3.3GHz/16GB).

This approach can be used (if necessary) to combine multiple partitioned files created by e.g. Spark engine.