Projects » Radio Times EPG
A Radio Times EPG for the Topfield TF5800 PVR
Nigel Whitfield, November 2005 (updated 27th January 2006)
This page describes how to collect data from the Radio Times web site and format it for uploading to a Topfield PVR, where it can be used as an alternative to the EPG broadcast via Freeview. The scripts were originally developed as part of a solution for Mac users, but can be used on Linux or UNIX based systems too.
Why use this system?
There are several reasons. Firstly the Freeview EPG is pretty skimpy, and doesn't always contain all the information you might want about a programme. Sometimes it's just simply not there (especially on ITV channels). Using an alternative data source means you can load more data, or more detailed information. You can also split that information between the standard and extended information - on Freeview there's usually no extended information, with everything crammed into the 'short description.' As well as making things needlessly cramped, API shortcomings on the Topfield PVR mean that TAPs can't usually retrieve all the text that is there.
So, by using information from the Radio Times site, you gain better quality information, including reviews of films details such as the director's name, along with Genre details, and you can split that between the short and long information fields, making better use of the on-screen displays. The downside is that you have to work to set things up, and it won't be updated in real time, which is possible (though not necessarily regular) with the Freeview EPG.
Outline and pre-requisites
To make a complete EPG system, you need a data source, a way to fetch that data, a means of formatting it how you want, and a way of storing it on your Topfield PVR. For this project, the Radio Times provides the data, and the EPG_uploader TAP provides the means to store it on your PVR. The code here handles the fetching of the data and the formatting. You will need a program to transfer it to your PVR for loading into the EPG by EPG_uploader; the tool you use will depend on your computer system. The scripts that form this project were written with Mac OS X in mind, but will run on most Linux/UNIX systems and, quite possibly, on Windows, though they've not been tested.
A complete solution for Mac users
If you're using a Mac, then things are very easy now; you need a program called MacTF-EPG, from Nathan Oates. It includes the scripts here, so they'll be installed along with the application; all you need to do is make sure you select UK from the locations in the preferences, and choose how you want it to upload the TGD files to your Toppy. Then, go to your home folder, then Library / Application Support / MacTF-EPG. You'll find the channelmap.ini file in there, and you need to edit it as described on this page, for the channels you want. You also need to configure the EPG_Uploader ini file. Once that's done, you just fire up MacTF-EPG, click the Fetch EPG Info button, and watch it create all the TGD files. You can browse them, search them, and even mark programmes to record, before clicking the Upload EPG to Topfield button.
Radio Times data
The Radio Times site provides a data feed consisting of 14 days EPG data for channels, which is available for personal use. This is used by projects such as XMLTV to provide programme guides for systems such as MythTV and many others, and it was my original intention to use the XMLTV format. However, a look at the data from the Radio Times site revealed that it is, essentially, a line of text for each programme, giving the EPG details, delimited by a unique character. Since the EPG Uploader format is much the same, I decided that passing through an intermediate XML stage was overkill.
EPG Uploader
The EPG Uploader TAP takes files in a format called TGD (fully documented in the readme), and stores them in the Topfield's Event Information Tables, blocking the loading of broadcast information if necessary. It is controlled by an epg_upload.ini that, while horribly complex in previous versions, is now much easier to configure, thanks to support for LCNs (logical channel numbers) in a recent beta. A sample configuration file appears below. If you don't already use the TAP and want to use the beta, you can download it here. Note that it is a test version.
How it works
The tar file that you can download contains six scripts and two sample ini files.
Additionally, the tar file will create two directories in the directory where you unpack it. The channels directory is used to receive the data from the Radio Times and the output directory is where the TGD files are stored.
To get started, download the tar file, and unpack it, then edit the ini files according to your needs.
Creating your channelmap ini file
The channelmap.ini file is responsible for controlling which channels you retrieve EPG data for. Lines that start with a # character are ignored; other lines should contain the name of a channel, followed by spaces or a tab, and then the id number from the Radio Times web site. The data file returned by http://xmltv.radiotimes.com/xmltv/channels.dat lists all the channels for which information is available, and their number. This number is used as part of the URL to retrieve the information; remember that for some channels, you need to ensure that you pick the right regional version, so you receive correct listings. You can call your channels anything you like - but no spaces in the names - though it obviously helps if the names are memorable. You need to use the same names in the ini file for EPG uploader, and you can also use the names as options to some of the scripts, if you just want EPG data for particular channels.
Here's an example channelmap.ini file, for the London area; as you can see, some channels have a # before them; these won't be retrieved.
# channel names here must match the ones you use in your # epg_upload.ini file # # main broadcasters first # remember to pick the correct region bbc1 94 bbc2 107 bbc3 45 bbc4 47 bbcnews24 48 itv1 26 #itv2 185 itv3 1859 itv4 1961 channel4 132 more4 1959 e4 158 e4+1 1161 five 134 # # UKTV # uktvgold 288 #uktvstyle 291 uktvhistory 801 #uktvbrightideas 923 #uktvfood 423 # # Sky # sky3 1963 #skynews 256 #skysportsnews 300 # # Other stuff # tcm 271 abc1 1856 discovery 147 # |
The channelmap.ini file should be in the same directory as the scripts.
Fine tuning the data you retrieve, via channelmap.ini
The latest versions of the script (version 0.5 and above) allow extra customisation via the channelmap.ini file, so that you can control exactly how much information each file gets. The included file contains a range of these options. Options in the file are specified by a space after the Radio Times id number, and then a series of parameters; each is the name, followed by an = sign and then the value. A comma (with no space) follows the option, if you want to specify another one.
The days= option lets you limit the number of days for a specific channel, like this, to limit UK TV Gold to 3 days of EPG data.
uktvgold 288 days=3 |
If the number of days is more than the overall limit, for example dat2tgd was run with -days 2, then the overall limit is the most that will be processed.
You can also limit the number of days of extended information for a channel, with the ext option
bbcnews24 48 ext=1 |
which would fetch only one day of extended information for News 24, but basic information for the same number of days as the other channels. You can combine options like this:
bbcnews24 48 days=3,ext=1 |
Additionally, you can limit the number of hours for which gather EPG data on a channel, using the hours and omit options. Both expect two times, and the first must be before the last, in the 24 hour clock. The hours option specifies the hours of data you want to grab, so for abc1 on Freeview, you'd say
abc1 1856 hours=0600-1800 |
The omit option lets you exclude programming between certain hours; use this, for example, for channels that start one day and end after midnight, such as UKTV Gold, which is billed as 1200-0500 on Freeview, like this:
uktvgold 288 omit=0500-1200 |
You can't specify both omit and hours on a channel, though you can combine these with other options.
The final two options allow you to restrict extended information to specific genres, either those that match with genres=, or those that don't match with except=. The information after the = is the genres, with multiple genres separated by |, for example
e4 158 except=soap |
to exclude descriptions of soap operas from the extended information on E4. You could include only drama and film on ITV2 in the next 5 days with this option:
itv2 185 genres=drama|film,ext=5 |
Creating your ini file for EPG Uploader
This used to be a rather complicated matter, since you needed to know the stream ids for each channel. The latest beta version, however, makes it much simpler, allowing you to specify an LCN for each channel, as in the example below. You need an entry for each channel for which you want to upload Radio Times EPG data. The service name can be anything you like - the only important thing is that it matches the names you use in the channelmap.ini file, which means that spaces are not allowed.
[General] GMT_Offset=0 Frequency File check=5 AutoLoadEPGdata=NO AutoDeleteEPGdata at start=NO Next Update Time=-1 TGD directory=ProgramFiles Load 99999999 event details=NO Display LOADING OSD=MINIMAL Logging Level (4everything 0nothing)=4 Broadcaster EIT=BLOCK EPG load wait=0 Timer Set Delay=38 LoadIceIni=NO Number of Services=4 [TV Parameters] [Channel 1] service_name=bbc1 LCN_channel=1 [Channel 2] service_name=bbc2 LCN_channel=2 [Channel 3] service_name=itv1 LCN_channel=3 [Channel 4] service_name=channel4 LCN_channel=4 |
Running the scripts
When you have created the ini files, it's time to run the scripts:
Mac OS X users
On Mac OS X, the built in Perl system doesn't include the module used to retrieve data from the web, so a PHP script is used; this has been tested on Mac OS X 10.4. All you need to do is execute the following commands at the command prompt, in the directory where you have stored the scripts:
php getdat.php ./dat2tgd.pl |
The first script fetches EPG data for all the channels you've specified in channelmap.ini, and stores it in the channels directory. The second converts those to tgd format, and stores them in the output directory, ready for upload to the PVR. The macepg.command script can be launched from the Finder, and will run both scripts for you automatically. Or, as described above, use MacTF-EPG to handle this for you, and provide a neat way to set timers on your desktop too.
Other systems
If you're using a standard Linux/UNIX system, your Perl installation probably includes the LWP::Simple module, so you can use the Perl getdat script, instead of the PHP version; if it gives an error, and you have PHP, try following the Mac OS X instructions above. The dat2tgd script has been successfully tested using Microperl on an Asus WL500g deluxe wireless router.
The two commands to run are:
./getdat.pl ./dat2tgd.pl |
If you're using an Asus router (probably an NSLU2 as well), the microperl installation won't run the getdat.pl script, since it doesn't have all the necessary libraries. To get round that, the downloadable tar file here contains a script called genwget.pl. Run this script after setting up or changing your channelmap.ini file, and it will generate a file called getdat.sh, which is a shell script version of getdat.pl that will fetch the files for you. There's also a simple sample script called tgdupload.sh, which uses puppy to copy the files to the Toppy.
./getdat.sh ./dat2tgd.pl ./tgdupload.sh |
Script options
There are some options to the scripts. Both perl scripts can be called with the -verbose option, so that you can see a report of what's happening, eg:
./getdat.pl -verbose |
The getdat scripts can both take a single channel name as a parameter, and will retrieve EPG data only for that channel, instead of all the ones you've listed in channelmap.ini, eg:
php getdat.php more4 |
Finally, the dat2tgd script can take a list of channels, and will produce output that includes only those channels. Here's what you'll see when you combine that with the verbose option:
./dat2tgd.pl -verbose itv4 sky3 |
A new option to the dat2tgd.pl script (December 2005, version 0.3) allows you to specify the number of days of EPG data you would like to retrieve; the Radio Times information is quite comprehensive, and if you retrieve it all, you will find that it can't be loaded into the Toppy, because it runs out of memory. Typically, when that happens, you'll see empty days at the end of your EPG. The new -days switch allows you to specify how many days worth of TGD files to produce (inlcuding today), and a line in the script can be edited to set the default.
Version 0.6 of the script adds a -help option to explain some additional options; type
./dat2tgd.pl -help |
to display it.
The default number of days for which to generate an EPG is governed by this section, near the top of the script file:
# $limit is the default number of days; use 0 for all available $limit = 0 ; |
Change the 0 for a different number of days; for example, 8 days to give you the same span of data as the Freeview EPG.
You can change the number of days when you run the script, like this, to get just three days of data:
./dat2tgd.pl -days 3 |
And you can, of course, specify channels too, after the number of days. If you want to have more days for some channels than for others - for example, you'd like 8 days for More 4 and BBC4, but only 3 days for the rest of your channels, then you need to create the larger number of days first, so you'd do this:
./dat2tgd.pl -days 8 more4 bbc4 |
If you did if the other ways round, the TGD files for the first 3 days would be overwritten by the ones that just had More 4 and BBC4 in them.
Transfer to the PVR
The TGD files need to be transferred to your Topfield PVR, and placed in the Program Files directory, where they can be read by EPG Uploader. The tgdupload.sh script will do this job on a Linux/UNIX system, using puppy.
Notes and credits
The EPG data that's provided by the Radio Times site is for personal use only. The code on this page was inspired by the Radio Times grabber code that forms a part of the XMLTV project. Tony's EPG_uploader TAP for the Topfield PVR also forms an essential part of this project.
None of this would have been possible without the work put in by the others. The code here may be distributed or modified freely.