Hack 81. Pulling Video from Your TiVo over FTP | Tivo Hacks. 100 Industrial-strength Tips Tools 2003

Hack 81 Pulling Video from Your TiVo over FTP

Extract recorded television shows from TiVo's Media File System using mfs_ftp and your garden-variety FTP application .

By this time you should be pretty comfortable FTPing [Hack #55] stuff back and forth to your TiVo. Most of your file transfers have, up to now, been one way and rather simplemoving small files and software to your network-enabled TiVo [Hack #49]. Let's raise the bar a little and siphon some of that recorded television programming out of the TiVo.

No matter how hard you look, you're not going to find those recorded television shows anywhere on your TiVo's filesystem [Hack #29]. The reason is simple: you're not seeing the entire filesystem.

TiVo's hard drives are organized in two ways: there's the standard Linux filesystem you are seeing when you telnet in [Hack #52], and then there is the Media Filesystem (MFS) that actually holds all the recordings. The regular old FTP server ( tivoftpd ) doesn't have access to the MFS, but Riley Cassel's mfs_ftp (http://alt.org/forum/index.php?t=getfile&id=81) does.

Installing mfs_ftp

Before getting mfs_ftp going, you'll need to get a hold of a few support programs: mfs_stdinsert , mfs_tarstream , mfs_stream , and mfs_export . Thankfully, all of these programs are available in one convenient package at http://alt.org/forum/index.php?t=getfile&id=86&rid=24.

Move the entire mfs_ftp_support_files.tar.gz archive over to your TiVo, drop it into /var , and unpack it:

 bash-2.02#  cd /var  bash-2.02#  gzip -d mfs_ftp_support_files.tar.gz  bash-2.02#  cpio -i -H tar -create-directories < mfs_ftp_support_files.tar  bash-2.02#  cd mfs_ftp  bash-2.02#  chmod 755 mfs_* ftpf

I know I've been making a habit of putting all our hacks into /var/hack , but the authors wrote this hack to be different and everything is based in the /var/mfs_ftp directory.

Once they're all across and in place, kill off your standard FTP daemon. There is no pretty, built-in way to shut down the FTP daemon, so you'll have to do it the Unix way. Figure out the process ID of the FTP server ( tivoftpd ) and kill it:

 bash-2.02#  ps auxw  grep tivoftpd  root   1220 0.0 0.0   0   0 ? SW  01:21  0:00 /var/hack/bin/tivoftpd  root   1233 0.0 0.0   0   0 p0 SW  01:21  0:00 grep tivoftpd bash-2.02#  kill -9 1220

You don't have to kill off tivoftpd ; you can run it alongside mfs_ftp . They serve different purposes and it may be useful to have them both running simultaneously . I suggest shutting down tivoftpd only so you can use the standard FTP port in the following examples.

Now start up the mfs_ftp daemon , invoking it on the command line, like so:

 bash-2.02#  /var/mfs_ftp/mfs_ftp.tcl 21 &

It'll start up, listening on port 21 (the standard FTP port) for incoming connections. If you have the tai l Unix utility installed [Hack #34], you can watch the msf_ftp server running, if you're so inclined:

 bash-2.02#  tail -f /var/mfs_ftp/ftp.log  00:00:00 - entering init_mfs_ftp 00:00:01 - entering background 00:00:01 - ping - mfs_ftp running on 127,0,0,1 : 21  00:00:01 - mfs_ftp version 1.2.5.9 - checking tivo system sw  00:00:01 - version 3.0-01-1-010  00:00:01 - setting mfspath to /Recording/NowShowingByClassic  00:00:01 - insert throttle is  0, == 0 ms  00:00:01 - extract throttle is 0, == 0 ms  00:00:01 - entering reset_mfs_ftp 00:00:01 - port 21 control socket initialized - ready for connections 00:00:01 - entering build_dir_LIST 00:00:01 - entering update_rec_fsids  00:00:01 - getting current rec_fsids list  00:00:01 - rec_fsids doesn't match cached_rec_fsids, deleting cached info 00:00:01 - entering build_rec_list  00:00:01 - from mfs "/Recording/NowShowingByClassic"  ....................................................................... 00:00:07 - build_rec_LIST complete

The tail utility will continue watching the ftp.log file, sending updates to the screen as it deals with FTP traffic. Stop tail at any time by pressing Control-C.

A Tour of mfs_ftp's Filesystem

Now that we have the mfs_ftp server up and running, use your favorite FTP program to FTP into your TiVo and take a look around. The directory structure of the server is as follows :

txt

Plain text files containing the same information as TiVo's Now Showing menu.

xml

Almost the same information as that found in the txt directory, this time in XML format. A typical file in this directorythis one representing a Friends episode ( 1686510 FriendsThe One With The Embryos.xml )looks something like this:

 <?xml version="1.0" tivoversion="3.0-01-1-010"?> <Object type="Recording" id="_top">  <SubObject type="RecordingPart" id="Part">   <Begin>0</Begin>   <CommercialSkipOffset>0</CommercialSkipOffset>   <End>1313822</End>  </SubObject>  <SubObject type="RecordingPart" id="Part">   <Begin>1314323</Begin>   <CommercialSkipOffset>0</CommercialSkipOffset>   <End>1798965</End>  </SubObject>  <RecordQuality>75</RecordQuality>  <SubObject type="Showing" id="Showing">   <Bits>35</Bits>   <Date>12198</Date>   <Duration>1800</Duration>  <Object type="Program" id="Program">   <Actor>AnistonJennifer</Actor>   <Actor>CoxCourteney</Actor>   <Actor>KudrowLisa</Actor>   <Actor>LeBlancMatt</Actor>   <Actor>PerryMatthew</Actor>   <Actor>SchwimmerDavid</Actor>   <AltEpisodeNum>466611</AltEpisodeNum>   <ColorCode>4</ColorCode>   <DescLanguage>English</DescLanguage>   <Description>Phoebe undergoes a procedure so she can serve as a  [RETURN]  surrogate mother.</Description>   <EpisodeNum>85</EpisodeNum>   <EpisodeTitle>The One With the Embryos</EpisodeTitle>   <ExecProducer>BrightKevin S.</ExecProducer>   <ExecProducer>KauffmanMarta</ExecProducer>   <ExecProducer>CraneDavid</ExecProducer>   <Genre>91</Genre>   <Genre>1002</Genre>   <GuestStar>RibisiGiovanni</GuestStar>   <GuestStar>RuppDebra Jo</GuestStar>   <IsEpisode>1</IsEpisode>   <NetworkSource>synd</NetworkSource>   <OriginalAirDate>10241</OriginalAirDate>   <Object type="Series" id="Series">    <Episodic>1</Episodic>    <Genre>91</Genre>    <Genre>1002</Genre>    <ServerId>16645</ServerId>    <ServerVersion>310</ServerVersion>    <ThumbData>268633087</ThumbData>    <Title>Friends</Title>    <TmsId>SH115127</TmsId>   </Object>   <ServerId>16805</ServerId>   <ServerVersion>42</ServerVersion>   <ShowType>5</ShowType>   <SourceType>2</SourceType>   <Title>Friends</Title>   <TmsId>EP1151270091</TmsId>  </Object>  <Object type="Station" id="Station">   <Affiliation>WB Affiliate</Affiliation>   <AffiliationIndex>53</AffiliationIndex>   <CallSign>WLVI</CallSign>   <City>Boston</City>   <Country>United States</Country>   <DmaName>Boston, MA-Manchester, NH</DmaName>   <DmaNum>6</DmaNum>   <FccChannelNum>56</FccChannelNum>   <LogoIndex>131303</LogoIndex>   <Name>WLVI</Name>   <ServerId>1387</ServerId>   <ServerVersion>16</ServerVersion>   <TmsId>11659</TmsId>   <ZipCode>02125</ZipCode>  </Object>   <Time>79200</Time>   <TvRating>4</TvRating>  </SubObject>  <StartDate>12198</StartDate>  <StartTime>79198</StartTime>  <StopDate>12198</StopDate>  <StopTime>80999</StopTime>  <StreamFileSize>720896</StreamFileSize>  <SubPriority>268633087</SubPriority>  <UsedBy>1</UsedBy> </Object>

As you can see, all the metadata about the television programincluding the episode description, actors, station on which it was broadcast, and oodles moreis encoded in this file.

ty

Recorded shows are stored as ty files, the most appropriate file type from which to extract the videos .

tmf

TiVo Media Format ( tmf ) files are similar to the ty variety and are particularly useful for extracting files with an eye to later reinserting them into TiVo.

tyx

Next generation tmf files, not yet ready for primetime because the software to use them is not as stable as one would like.

shutdown

More of a trigger than a directory, the very act of navigating into this folder will shut down the mfs_ftp server.

phoenix

Like the shutdown directory, moving into the phoenix directory will restart the server (read: kill off mfs_ftp and start a new version in its place). Think "phoenix rising from the ashes"geek humor again, I'm afraid.

In actuality, these directories are created on-the-fly by the mfs_ftp daemon for your convenience, making browsing the contents of the filesystem that much easier. The real MFS is not packaged nearly as nicely .

Transferring Video Files

Since our goal here is to grab video files over FTP, switch to the ty directory and download a television program you're interested in.

The mfs_ftp server supports only one connection at a time. Feel free to create a queue of television shows and configure your FTP client to grab them serially , but don't expect to be able to download more than one program at a time.

Extraction speed can vary quite a bit; some users have reported speeds as slow as 20 KB per second (quite slow and probably a sign of defective hardware), while others have been lucky enough to reach speeds of 1 MB per second and faster. About the only factor you can influence is the speed of the Ethernet connection; I recommend that Series 1 users get their hands on a TurboNET adapter [Hack #49], with its onboard 100 megabit Ethernet port, and extract either directly to a 100 megabit-enabled PC or Mac or over a 100 megabit network.

Go To

Those extracted ty files aren't going to do you much good without a media player [Hack #83] to read them or some way to convert them [Hack #85].

Top