Reducing memory usage of FOP

Here is a straight forward example on using FOP to generate PDF reports:

......
<fo:page-sequence master-reference="simpleA4">
......
  <fo:flow flow-name="xsl-region-body">
    <xsl:apply-templates select="the-bean" />
  </fo:flow>
......
</fo:page-sequence>
......

The original data is in form of Java beans. A container contains multiple Java objects of "TheBean". The data is converted into XML by Castor. You can notice that the XSL file simply "apply-templates" based on the beans. FOP is smart enough to loop through all the beans of type "TheBean" to render the reports.

But this straight forward approach has a problem. FOP writes output and free up resources on a per-page-sequence basis. In the above example, the "apply-templates" tag is within a page sequence. This means FOP will loop through all the data before wriing the output PDF. In one particular test, in order to generate a report with 11MB of data in XML format (the PDF output is over 400 pages), FOP used up 1.3GB of memory.

To get around the problem, we could use a difference page sequence for each object:

......
<xsl:for-each select="the-bean">
  <fo:page-sequence master-reference="simpleA4">
......
    <fo:flow flow-name="xsl-region-body">
      <fo:block>
        <xsl:apply-templates select="." />
      </fo:block>
    </fo:flow>
......
  </fo:page-sequence>
</xsl:for-each>
......

Here, the for-each loop will iterate the beans one by one using different page sequence. Using this approach on the same 11MB XML file, the memory usage is reduced to below 48MB.

If each object needs to maintain its page number count, we could use a variable to name the "last page block" differently:

......
<xsl:for-each select="the-bean">
  <fo:page-sequence master-reference="simpleA4"
    initial-page-number="1"
    force-page-count="no-force">

    <xsl:variable name="count">
      <xsl:value-of select="position()"/>
    </xsl:variable>
......
      Page <fo:page-number/> of
      <fo:page-number-citation
      ref-id='last-page{$count}'/>
......
    <fo:flow flow-name="xsl-region-body">
      <fo:block>
        <xsl:apply-templates select="." />
      </fo:block>
      <fo:block id="last-page{$count}"/>
    </fo:flow>
......
  </fo:page-sequence>
</xsl:for-each>
......

Note the use of force-page-count="no-force". It is to prevent FOP from inserting a blank page between page sequences.

20071116 馬灣公園

行呀行

之前話過每個禮拜都要出去行下運動下，不過上個星期偷懶…好在今朝戰勝睡魔。出發時天色陰暗，仲以為會落雨。點知愈行愈好天，仲有涼風，行得好鬼舒服！

是日金句

十一月十一日，1111，四條光棍，係大陸的「光棍節」。大陸人好怕單身咩？！


剛見識到的「光棍節」名句： 『一個人吃飽，全家不餓』。 係單身的無奈，還是好處？ =.=

因為豆漿濃啊

網上論壇新興的「無厘頭」回覆… 『因為豆漿濃啊！』

=.=...

又係十下十下呀！

今晚又係同朋友打邊爐… 應該係連續五個 weekend 了… =__=...

不過今次有d唔同：第一次有七個人咁多一齊打，差d唔夠椅子添…真係唔好意思 :p 。其次係我第一次食大閘蟹！唔係講笑的。不過原來我屋企人唔食大閘蟹，因為個個都係寒底一事係堅嫁！我食完真係有d暈暈地… @.@… (不過亦都可能係我飲得太多 :P)

PS. 多謝大家俾面上來玩！

是日金句

「本來諗住關心下佢… 點知越講越衰」

12 years

what can you accomplish in 12 years?

DVD authoring

Recently downloaded a video in MKV format. MKV is quite a popular format when distributing video contents. One of the reasons is that MKV is extremely flexible. It can virtually mix any video/audio content in it. But this also posts a challenge when converting it.

First, let's check what is in the MKV file so that we know what to do. mkvtoolnix is a handy tool for that. Use the mkvmerge command to see the file structure:

mkvmerge -i 5Centimeter.mkv

And this is the output for my MKV file:

File '5Centimeter.mkv': container: Matroska
Track ID 1: video (V_MPEG4/ISO/AVC)
Track ID 2: audio (A_AC3)
Track ID 3: subtitles (S_VOBSUB)
Track ID 4: subtitles (S_TEXT/ASS)

Here is the analysis:

• The first track is the video in MPEG4 format. We will need to convert it to MPEG2 for DVD


• The second track is the audio in AC3 format. We can just reuse it in the DVD. No conversion required


• The video is a Japanese anime and track 3 is the subtitle file


• Someone is nice enough to include a translation of the subtitle in Simplified Chinese format in track 4. But note that it is in text format . So we will need to convert it into image format for the DVD

Initially I thought I could use avi2dvd to do most of the conversion jobs since it supports MKV format and subtitle generation etc. But after several days of struggle, I concluded that it can't handle Chinese subtitle nicely. So instead, I just used it to do the video and audio conversion, format/frame rate changes, pull down etc tedious tasks and generate a "DVD ready" ISO for me, without the subtitles.

avi2dvd is actually an GUI for many freeware programs. There are many options to choose from. BTW, I used the HCenc encoder, which is slow (almost 3 hours to encode the 1 hour video on my 3200+) but is supposed to give good result.

Once we have the ISO file, we need to demux it. This is to get the converted video (and audio) files so that we can multiplex them with the subtitles. I used VobEdit for that.

The subtitle requires lots of work here. First, use mkvmerge to extract the text from track 4 into a file. It is in SRT format with GB encoding. As I couldn't find a freeware tool that can handle SRT directly for DVD authoring, so here is the process on how to convert it into SUP format:

• Since I prefer to read Traditional Chinese, so I used ConvertZ to convert the content into Big5.


• Use Subtitle Workshop to convert the SRT file into Substation Alpha (SSA) format





• Use MaestroSBT to convert the SSA file into SON, with the text rendered as bitmap. Choose the output to be 4-bit compressed bitmap files. Since I am rendering Chinese text with Big5 encoding, I changed the default font setting. Also need to adjust the margin etc. In the Color Rendering section, I chose "Two colors for text, one for outline, no antialias". Note down the color used. We will need this info later.





• Use son2vobsub to convert the SON file to SUB format


• Use SubToSup to convert the SUB file into SUP format

... and finally... we have a usable subtitle file... orz...

Next, use IfoEdit for the final mutliplexing. Choose "DVD Author" -> "Author new DVD". Select the video, audio and subtitle files. Set the correct language for the audio and subtitle. Then click OK to generate the DVD files (those VOB, IFO etc files).

Then a little trick. Since the subtitle is generate in specific colors, we need to set it in the IFO file so that it can overlay nicely on the video. In IfoEdit, select the "VTS_PGCITI"->"VTS_PGC_1" node. Then look for the "Color 0 Y Cr CB", "Color 1 Y Cr CB" etc entries. They are the color codes for the subtitle:

Color 0 = Background
Color 1 = Outline
Color 2 = Letters
Color 3 = Anti-alias

Here are the codes for different colors:

White: EB 80 80
Brown: 30 B8 6D
Black: 10 80 80
Green: 50 50 5A
Oliv: 71 89 47
Naval: 1C 76 B8
Púrple: 3D AF A5
Turquese: 5D 47 92
Grey: 7D 80 80
Silver: B4 80 80
Red: 51 EF 5A
Lime: 90 22 35
Yellow: D2 92 10
Blue: 28 6D EF
Fucsia: 6A DD CA
Aqua: A9 10 A5

Change the values to reflect the colors used when we generate the subtitle images using MaestroSBT. Click "Save" to update the IFO. Then click "Disc Image" to generate the final ISO.

Preview the ISO. To get the position right, I needed to re-generate the subtitle file a few times... orz. When everything is OK, use a burning program (e.g. DVD Decryptor) to burn the image to a DVD-R... and we are done!

Encoding video for N80

The Nokia N80 has a bright and (relatively) high resolution screen (352x416). But unfortunately, the RealPlayer on the phone can't play MP4 streams at the max native resolution. After some trail-n-error and some googling, here are the settings for encoding video for N80:

• File format: MP4


• Video codec: MPEG-4


• Audio codec: AAC


• Resolution: 352x288