ureadit

Purpose

Use "cue points" embedded in a Flash video to display foreign language captions

Motivation

Educational and informative videos are being created and deployed on the web for general consumption by a world-wide audience. However, language barriers can exclude large groups of people if the video is not translated into the local language. The translation process can be tedious and difficult, as the translation - whether audio clips or captioned print - must remain synchronized with the video.

Conveniently, Adobe Flash videos offer the ability to insert "Caption Points" into the video so that external captioning processes can trigger audio or visual aids that will help those that don't understand the original language of the video. There are thus two aspects to providing sufficient translation: 

1. Translation of materials must be tied to time-stamps in the video file, where caption points with unique identities can be used to trigger external events.
2. The viewer selects the proper translation type for his or her local language or dialect.

This project helps simplify both aspects of translation by providing tools that will simplify the translator's job in creating the translations and providing simple XML files so that the viewer can simply "plug in" his or her localized XML file to get the translation. With this Flash tool, a single Flash video file (FLV) can easily be translated to any language with either written or audio translations. 

Current Status

July 31, 2009

Project started on July 31, 2009. Basic FLV test file produces, and Cue Point events are being handled properly. Translation in both English and Japanese are available in separate XML plug-in files. Format of the XML plug-in files has been set.

August 2, 2009

Created code to read a "Master" XML file into the application. The master XML file contains a pointer to the video file as well as all the language translations available. The language translations will be represented by small icons with the flag of the nationality of the language being translated. These buttons - with the flag icon faces - will have to be built dynamically as the master XML file is read in. This is more of a pain than I originally anticipated, as I need to build an iterating script that reads in each nationality, the flag and the file representing the translated text. I'm thinking now of extending the Button class to add a few more static items, like a pointer to the icon file and a pointer to the XML translation file. In that way, the applcation could just call a fresh button and hand it pointers to the flag icon and the XML file location, and the button would build itself. Oh ... it would need a location to place itself, so I'll create a Sprite on the main application window that will contain all language buttons.

BTW: I found an excellent site to get all flags for all countries recognized by the United Nations.

The code below shows what happens after the master XML file is loaded. I'm parsing through the XML trying to extract the name of the movie (only one movie per master file) and the different languages for translation.

function loadedVideoXML(evt:Event):void {
 vxml = new XML(ul.data); // create the xml object
 var movie:XMLList = vxml.elements("movie");<
 if(movie.length() != 1) {
  captionText.text = "Error - Malformed XML document. One and only one \"movie\" element must be present";
 } else {
  movieFile = movie[0].attribute("filename");
  for each(var xmovie:XML in movie[0].child("translation")) {
   translationFile.push(xmovie.attribute("xmlfile"));
   // STOP HERE: rather than using an array to hold the XML file information,
   // I think I'll just extend the Button class and create each button with<
   // attributes from this XML file. Then all we have to do is create a new
   // button and give it the translation XML file location, a pointer to the
   // appropriate flag button, the container Sprite where it should appear on
   // the main application window, and we're done. The button will capture its
   // own clicks and start up the translation process using the file name it was
   // given. Hmmm ... okay, this will take a couple more hours, so I will stop
   // now. Bed time.
  }
  if(movieFile == "") {
   captionText.text = "Error -  Malformed XML document. Movie file not specified.";
  }
 }
}

 August 4, 2009

Created the button, but used a Sprite class instead. Sprites are a lot easier to deal with. This class acts like a normal button, except that it can hold a file name for the associated XML translation file. So when the mouse clicks on it, it sends out an event that is trapped by the main routine, and then the main routine can inquire the button that generated the interrupt and find out what the XML file name is.

Here's the button/Sprite class:

  public class TranslateButton extends Sprite {

private var xmlFile:String;

private var language:String;

private var loader:Loader = new Loader();

public function TranslateButton(xf:String, cf:String, lan:String):void {

xmlFile = xf;

language = lan;

loader.contentLoaderInfo.addEventListener(Event.COMPLETE,loadedHandler);

loader.load(new URLRequest(cf));

alpha = 1.0;

}

private function loadedHandler(evt:Event):void {

var sw:Number = 23.0/loader.width;

var sh:Number = 15.0/loader.height;

var matrix:Matrix = new Matrix();

matrix.scale(sw,sh);

var bitmapData = new BitmapData(23,15);

bitmapData.draw(loader,matrix);

this.graphics.lineStyle(1,0x808080);

this.graphics.beginBitmapFill(bitmapData);

this.graphics.drawRect(0,0,bitmapData.width,bitmapData.height);

this.graphics.endFill();

}

public function getXMLFile():String {

return xmlFile;

}

public function getLanguage():String {

return language;

}

}

}

I've decided to leave the "alpha" variable hanging there, with a possible option to tweak it a bit for aesthetic purposes. Same goes for the graphics.lineStyle -- a black line around the flag looks too intrusive, so I made it a light gray. I used a Matrix class to scale the flag to a size that is consistent with what is needed; however, I still need to play with the numbers a bit. I'm not too happy with the overall aethetics, but I'm getting close.

 August 5, 2009

I've pretty much completed the prototype. Once the flag buttons were working, the rest fell into place quickly.

From an operational perspective, the program works like this: 

1. When the Flash application starts, the "video.xml" file is read in. This file, which is in XML format, identifies the movie to start playing. This XML file also identifies the language interpretation files that are available. These "interpretation" files are also XML files that have children, where each child has a cue-point attribute and data associated with a foreign language phrase.
2. Until the user hits a "flag" icon, there will be no translation. When the user hits a flag icon, it creates an event that is captured by Flash.
3. Flash looks into the event caused by the hitting of the flag icon, making an inquiry to the icon itself and asking it to identify its associated translation file.
4. Once the translation file is identified, the Flash application loads that file, which - as mentioned before - is in XML format. An internal XML structure is used to hold the contents of the file. This XML structure holds all the translated information for the language associated with the icon.
5. When the file plays, it will eventually hit a cue-point. This creates an event that is captured by the Flash player.
6. The Flash player looks at the name of the cue-point and then searches through the language XML structure to find a child with an attribute that matches the name of the cue-point.
7. Once this child is found, the Flash application loads its associated data, which will be the translated text corresponding to the cue-point.
8. The translated text is sent to the caption box.
9. If the user manually moves the player head, the player will send a "seeked" event. The Flash application will then erase any text in the caption box so as to not mislead the user with the incorrect translated text. When another cue-point is encountered, the appropriate translated text will be displayed.

August 6, 2009

Investigated placing a back-end on this application. Here's the problem: A translator must transcribe the original video in its native language, then place cue-points in the FLV file, then create the tranlated text, then put this translated text into the XML file. Translating text is difficult enough, but being forced to put this text directly into an XML structure constitutes torture! What I envision is a tool that allows you to run a video and hit the pause button after a few phrases are spoken. Then you can simply enter the translated text corresponding to those phrases directly on the screen. Then you resume the video and go on to the next phrase. Behind the scenes, the back-end software will take care of inserting the cue-points, naming them, and then putting the translated text into the XML file.

What function does this? I'm glad you asked! Within the FLVPlayback class, there is a method called AddASCuePoint. This method does not write a cue point into the FLV video file; rather, it creates a set of cue points that get attached to the video dynamically when the video is loaded. It's not quite as good as putting the cue points directly in the FLV file, but it's the next best thing. I'll simply have to create YAXMLF (Yet Another XML File) that will hold information to dynamically hold the cue point information and be loaded at the same time that the FLV video file is loaded.

The only remaining problem is actually writing the XML files onto the server. Hmmmm ... not sure how to do this, other than asking the web-master to upload all these files when the video is loaded onto the website. Perhaps Adobe AIR might be the right choice.

Stay tuned! 

 August 9, 2009

Well, I've looked into creating an Adobe Air backend application, and I've deemed this to be too much trouble for now. Apparently Air does not provide some of the more automatic dialog boxes for opening and closing files, save, save-as, etc. These are the most basic things that I would have expected, and I don't mind writing them myself, but at this point, it would be too time consuming. For the time being, I'll leave it to the user to create his/her own XML language files.

For the time being, I'll package up what I have and place it on the "Product" page, as soon as I create the product page. This will allow everyone to have a look at it and provide me with feedback. If there's enough interest in it, then I'll create a back-end for it. :-)

Meanwhile, I'll continue with a video carousel project I've had on the back-burner for a while.

 August 13, 2009

Okay, so I've been forced to revisit this project because we're finding out that it's just too difficult to manually put cuePoints into FLV files. In lieu of putting cuePoints directly into FLV files, I added some software that allows you to put cuePoints into an XML file. The cuePoints are then extracted from the XML file and put into the FLV file at runtime. It works pretty well, and it makes things a bit easier, but still too difficult for the average person. So now I'm stuck with having to build up the back-end to this product. Further, it MUST be done using either Flash itself or Air, because it needs to interact with the FLV file as the user puts in the cuePoints. I know I'm not making much sense at this point, but suffice it to say that I have to get back into this project because the people that need to use it do not have the sophistication to do everything without a lot more code.

So tonight I started building the Air application for the back end. I'll keep you posted.

August 24, 2009

Working on this project on an as-available basis. Using Adobe AIR and am fairly impressed with it so far. The only problem I'm having is attemping to inject cue points into a movie while it is playing. Using the FLVPlayer.addASCuePoint call, but the loaded movie doesn't seem to understand all the cue points. I'm sure it's a simple bug. I'll look at it next.

Meanwhile, the "back end" development is looking fairly good. It's simple to build a GUI using AIR. I remember the "old" days when creating Windows software using Borland's Turbo C++ (circa 1994). What a pain it was to build a GUI and create the "callback" functions. Nowadays, it's pretty much drag-n-drop your components onto the GUI, name them and write their event functions. What a breeze! :-)

Stay tuned ...