Streaming large (binary) data

[Jörg Schaible]

Hi Martin,

sorry for the long delay. However, you should use the user's list for
questions.

Hi,
I have to export lots of objects from a database into a XML file. These
objects contain large BLOBs, so I started to implement a "BlobConverter".
public void marshal(Object source, HierarchicalStreamWriter writer,
MarshallingContext context) {
Blob blob = (Blob) source;
context.convertAnother(blob.getBytes(1, (int) blob.length()));
This would read the whole BLOB into a byte array and delegate to
EncodedByteArrayConverter, which in turn would write a base64 encoded
string. This creates two arrays in the heap (the byte array and the char
array with the base64 encoded data), which are way too large to fit into
the available heap space.
So I am looking for a solution which would allow to stream the binary data
into a streaming base64 encoder and to stream that into the output stream.
However, I could not find any method in HierarchicalStreamWriter which
would allow to stream character data. There is just "void setValue(String
text)", which requires the whole data to be in the heap.
Is it possible to stream character data to the output stream?

XStream has no direct support writing or reading an element's text with
streams (XSTR-532). However, although outside of the XStream specification,
it should be possible to call setValue multiple times for stream-based XML
drivers. Therefore you could use a Base64 stream (e.g. from Apache commons-
codec) to write chunks into a buffer and write that one with setValue before
continuing with the next chunk.

- Jörg

[Tatu Saloranta]

One related thing worth noting is that Woodstox and Aalto Stax parsers implement Stax2 extension API, and support both typed access to base64 encoded content, and allow that to be done incrementally (block by block). Both for reading and writing. So this is a very efficient way of dealing with large binary content in this context (probably more so than trying to use attachments).

Interfaces used are XMLStreamReader2 / XMLStreamWriter2, implemented by readers/writers that Woodstox and Aalto return.

-+ Tatu +-


-----


On Friday, September 26, 2014 12:27 AM, Jörg Schaible <joerg.schaible-***@public.gmane.org> wrote:



Hi Martin,

sorry for the long delay. However, you should use the user's list for
questions.

Hi,
I have to export lots of objects from a database into a XML file. These
objects contain large BLOBs, so I started to implement a "BlobConverter".
public void marshal(Object source, HierarchicalStreamWriter writer,
MarshallingContext context) {
Blob blob = (Blob) source;
context.convertAnother(blob.getBytes(1, (int) blob.length()));
This would read the whole BLOB into a byte array and delegate to
EncodedByteArrayConverter, which in turn would write a base64 encoded
string. This creates two arrays in the heap (the byte array and the char
array with the base64 encoded data), which are way too large to fit into
the available heap space.
So I am looking for a solution which would allow to stream the binary data
into a streaming base64 encoder and to stream that into the output stream.
However, I could not find any method in HierarchicalStreamWriter which
would allow to stream character data. There is just "void setValue(String
text)", which requires the whole data to be in the heap.
Is it possible to stream character data to the output stream?

XStream has no direct support writing or reading an element's text with
streams (XSTR-532). However, although outside of the XStream specification,
it should be possible to call setValue multiple times for stream-based XML
drivers. Therefore you could use a Base64 stream (e.g. from Apache commons-
codec) to write chunks into a buffer and write that one with setValue before
continuing with the next chunk.





- Jörg


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email