Some hints for getting free-to-air satellite channels into the Electronic Program Guide in Kodi (or another frontend)

If you are running a satellite backend system such as Tvheadend or MediaPortal (or MythTV, if you are one of the lucky few that can actually get it to work), and you use Kodi or the MythTV frontend, then it is possible to populate the schedule grid with listings from many sources. Note I did not say that it is easy, just that it is possible. The key is to use an external program such as zap2xml (Zap2it TV listings to XMLTV or XTVD .xml) or zap2epg – xmltv grabber for USA and Canada. These are commonly referred to as “schedule grabbers”, or just “grabber” programs.

The real trick is figuring out how to use one of those programs. Typically they are used to grab listings for a single over-the-air market, not a hodgepodge of stations and services from various locations. Such programs will create a xml file that contains schedule listings (in Tvheadend it will typically be at /home/hts/.xmltv/tv_grab_file.xmltv, assuming that “hts” is the Tvheadend username on your system), and that file will have all the over-the-air channels in your area, or all the cable or satellite channels from your provider. If you want to use zap2xml and you’ve never set it up before, I’ll give you some setup hints later in this article. But for now, lets assume that you have it all set up and you know how to create an xml file (named /home/hts/.xmltv/tv_grab_file.xmltv) containing your local listings.

Once you have created the xml file, it can be imported into the Tvheadend, MediaPortal, or MythTV database. There are now two different methods for importing channels into Tvheadend, a newer method that uses a socket, which I will discuss momentarily, and an older method that uses a file called tv-grab-file, which must be downloaded and moved into the /usr/bin directory on the system running Tvheadend (be sure to make it executable, since it is a bash script). Also, you may need to edit the line in the script that starts with “cat” and contains the path to the tv_grab_file.xmltv file, to specify the correct path and file name of the file produced by your selected grabber program, if it isn’t being saved as /home/hts/.xmltv/tv_grab_file.xmltv. Note that if the tv_grab_file.xmltv file is not owned by the Tvheadend user, it must at least be made readable by Tvheadend – incorrect permissions and/or ownership on this file will make it inaccessible to Tvheadend.

Note that in the newest versions of Tvheadend, the grabber settings have been broken up into three tabs, labelled “EPG Grabber Channels”, “EPG Grabber”, and “EPG Grabber Modules”, but you may not be be able to see all of the grabber settings until you do this: Click on the Configuration tab, then the General tab, then the Base tab, and then in the “User interface level” setting choose “Expert” from the dropdown. Then click “Save” (directly under the “Base” tab). Now you should be able to see all the EPG Grabber settings. In this new interface, you select the grabber in the “EPG Grabber Modules” tab, and all of the grabbers listed there should be disabled (orange dot at start of line) except for either “External XMLTV” (if you plan to use the socket method) or “Internal XMLTV: tv_grab_file is a simple grabber …”. Whichever you select should have a green dot at the start of the line (click on any dots that are the wrong color to change them, then don’t forget to click the “Save” button). This screenshot shows tv_grab_fle selected:

Once you have done that, you tell Tvheadend to use the tv_grab_file script by going to Configuration | Channel / EPG | EPG Grabber page. First, if any grabbers are enabled in the “Over-the-air Grabbers” section, disable them by unchecking the boxes next to them. Then, in the “Internal Grabber” section, select “XMLTV: tv_grab_file …” in the dropdown (you may need to restart Tvheadend or reboot the system before it will appear – if it still doesn’t appear, check the ownership and permissions of the file, it should be the same as the other tv_grab_* files in the /usr/bin directory – owned by root, and executable by all users). This is how it looked in previous versions of Tvheadend:
In the newest versions of Tvheadend you can actually schedule the Internal Grabber to import the listings at a specific time each day. Here I have set it to run at 2:33 AM every day (after commenting out the default):

You set the time you want the grabber to run in the “Internal grabber” section of the “EPG Grabber” tab. I will spare you my usual rant about developers making changes in the user interface that make things more confusing for users, but suffice it to say that it appears that every new version of Tvheadend changes the user interface in significant ways, so you may have to look around a bit to find these settings.

There are two other ways to get Tvheadend run the grabber.  One is found in newer versions of Tvheadend, where in addition to the “Trigger OTA EPG Grabber” button shown above, there is also a button labelled “Re-run Internal EPG Grabbers”.  If you click that button, it will immediately read the file containing the listings into Tvheadend.  The other way is to run a command from the Linux command line or from a shell script that looks like this:

/usr/bin/curl 'http://user:pass@localhost:9981/api/epggrab/internal/rerun?rerun=1' >/dev/null 2>&1

Please make certain the single quotation marks (‘) are really single quotes and not “prettified” quotes (WordPress sometimes changes them).  Replace “user” and “pass” with the username and password you use with Tvheadend.  And if your installation of Tvheadend runs on something other than port 9981 (which is the default) then you’ll need to change that also.

If you would prefer to try the newer method that uses a socket, then in the “EPG Grabber Modules” tab you would disable any other grabbers and enable only “External XMLTV” by checking the “enabled” box in the screen shown below (then, don’t forget to click the “Save” button). But make a note of the path shown in the “Path” field; you will need to know that later.

Now run the following commands from a Linux command prompt – the responses you get will determine which method you use to import the guide data into Tvheadend:

which socat
which nc
which curl

Ignore the ones that do not return a path to the program mentioned. For example, typically if you run “which nc” you will get a response such as /bin/nc – that indicates that the “nc” (netcat) program is installed on your system.

If typing “which socat” returned a path, then you can use this to import your guide data into Tvheadend using the socket method, replacing the paths and filenames as needed:

cat /home/hts/.xmltv/tv_grab_file.xmltv | socat – UNIX-CONNECT:/home/hts/.hts/tvheadend/epggrab/xmltv.sock

If typing “which nc” returned a path, then you can use this to import your guide data into Tvheadend using the socket method, replacing the paths and filenames as needed:

cat /home/hts/.xmltv/tv_grab_file.xmltv | nc -w 5 -U /home/hts/.hts/tvheadend/epggrab/xmltv.sock

If neither “which socat” nor “which nc” returned a path, but “which curl” did, then as a last resort you can use this to import your guide data into Tvheadend using the socket method, replacing the paths and filenames as needed:

cat /home/hts/.xmltv/tv_grab_file.xmltv | curl -d @- -m 5 -X POST –unix-socket /home/hts/.hts/tvheadend/epggrab/xmltv.sock http://127.0.0.1

Whichever of the above three lines you use, note that you may need to run it as the hts user or you may encounter permissions or ownership issues. Replace “/home/hts/.xmltv/tv_grab_file.xmltv” with the path and filename of your xml file that contains your local listings, and replace “/home/hts/.hts/tvheadend/epggrab/xmltv.sock” with the path that was displayed in the “Path” field of the External XMLTV grabber in the “EPG Grabber Modules” tab (the one I mentioned that you would need to know a few paragraphs above).

If you can’t get this to work then all I can suggest for now is using the older method described earlier. But if you can get it to work, there are a few advantages to using a socket. For one thing, you can import your listings directly into Tvheadend in whatever script you may be running to get your listings – there will be no delay between the time you get your listings and the time they are imported into Tvheadend. And if you have to do an extra download of schedule data for some reason (say you add a new channel) the listings will be imported immediately.

After you get the schedule grabber working, all you need to do set up a cron job or scheduled task to a run your listings grabber (the zap2xml program, or whatever you use), or to run a bash or shell script that runs your listings grabber, once a day. Then if you are using the older method to import listings into Tvheadend as shown above, set it up to import the listings fifteen minutes after you have run the listings grabber (you could probably get by with a shorter interval, but why rush it – you want to make sure the listings grabber has completed its task before Tvheadend grabs the resulting file). If you are using the newer socket method, just include the “cat” command that uses socat, nc, or curl at the end of your bash or shell script.

Don’t be tempted to comment out or delete all the entries in the “Cron multi-line:” textbox in the Internal Grabber section of the configuration.  I tried that and found that the grabber ran every two minutes!  So leave at least one of the entries uncommented, whether it be the default entry or one you have added.

NOTE: There are some people that find that for whatever reason, they cannot run the tv_grab_file script. This most often happens on systems or devices where bash is not installed (to determine whether that is the case, enter which bash at a Linux command prompt and if bash is installed it will display the path, typically /bin/bash). In such a case it may be possible to use a modified tv_grab_file script. For example, in my Review of the TBS MOI+, I showed a variation of that script that runs under ash (as provided in BusyBox), since the original bash script won’t work in that environment. But if you can’t do it that way, try using the newer socket method described above. That method is also discussed in this thread on the Kodi forum.

Note that after Tvheadend imports NEW channels, you MUST refresh the browser window before the new channels will appear in the EPG Source dropdowns under the Configuration | Channel / EPG | Channels tab. You may even need to close and re-open your browser. Failure to do this is probably the #1 reason people think Tvheadend has not imported the newly-added channels. You must select an EPG source for each channel before schedule data for that channel will be read into Tvheadend, which will happen on the next scheduled import of the TV schedule data.

NOTE: If you are using the older method (the tv-grab-file grabber), you can force an unscheduled read of TV schedule data in the newest versions of Tvheadend by clicking the “Re-run Internal EPG Grabbers” button under the Configuration | Channel / EPG | EPG Grabber tab. If you have an older version of Tvheadend that doesn’t have that button, you can accomplish the same thing by temporarily setting the Internal Grabber Module to “No grabber”, clicking the “Save Configuration” link, then changing the Internal Grabber Module back to “XMLTV: tv_grab_file …” and clicking the “Save Configuration” link again. Or as previously noted, you can run this from the Linux command line or from within a shell script:

/usr/bin/curl 'http://user:pass@localhost:9981/api/epggrab/internal/rerun?rerun=1' >/dev/null 2>&1

Again, please make certain the single quotation marks (‘) are really single quotes and not “prettified” quotes;  replace “user” and “pass” with the username and password you use with Tvheadend; and if your installation of Tvheadend runs on something other than port 9981 then be sure to change that also.

If you are using the newer socket method, just re-run the “cat” command that uses socat, nc, or curl to re-import your data to Tvheadend.

When you are setting up your listings sources (channels) on whatever TV listings service you plan to use, if you don’t know which providers or stations carry a specific channel, you can look it up on Wikipedia, which will often tell you which providers and/or local stations carry that channel.

There are some national services that I will not name here, but that aren’t listed in the channel listings because they aren’t intended for viewing by home viewers. You need to get creative with those. For example, if you just happen to find a feed of the QXZ network, and you are smart enough to not blab about it all over creation so that the signal gets scrambled, you may be able to get listing data for at least prime time by grabbing the listings for the network owned “flagship” station in your time zone, which might be WQXZ on the east coast or KQXZ on the west coast. Of course the QXZ network is totally fictitious, but hopefully you get the idea.

If you spend a little time studying the XMLTV File format, you can even write your own scripts or programs to create “fake” XMLTV data for certain stations – I suppose you could even do it manually if you are a very patient and precise person.

There is one pitfall to all this, which was actually more of a problem with a grabber program that’s no longer useful due to changes in the underlying service, but I’ll mention it anyway just in case you ever run across it. The various schedule sources sometimes change channel ID numbers without any advance warning, particularly when you are grabbing terrestrial (over-the-air) channels. If you find that the schedule data for a particular channel “runs out” after a certain day, it’s probably because the ID numbers have changed. In Tvheadend, go to the Configuration | Channel / EPG | Channels tab, find the affected channel(s), and change the EPG Source using the dropdown – you will probably see both the old and new ID’s, usually one underneath the other. Uncheck the old one and check the new one, and the next time Tvheadend updates its EPG information, it should be okay (don’t forget to click the “Save” button before you leave the page!). By the way, in case you hadn’t figured it out from the previous paragraphs, that’s the place where you select the channel data to associate with a channel in the first place, but I will again note that if the EPG sources aren’t appearing in the dropdowns after Tvheadend has imported your xml file, then you may need to refresh the page, or close and re-open your browser.

EDIT: For those that have never set up zap2xml before, here is the general procedure. These are basically their instructions, but with some added comments to help clarify what needs to be done.  If you prefer to use zap2epg, please see the instructions in their wiki.

EDIT (January 2018): Unfortunately, Zap2it has changed the format of their web interface. The latest releases of zap2xml (dated 2018-01-10 and later) do still work with both Zap2it and TV Guide, but you will need to download the latest version, and be sure to read the 2018 release notes. The instructions have been changed to reflect the changes in the setup procedure. When using either Zap2it or TV Guide as your listings source, you definitely should set your favorite channels, but if using Zap2it you will want to note the new -8 option in the zap2xml release notes, while if using TV Guide I have found that favorites may not “take” on your first attempt to set them – once you log out and back in, they just might be gone, and you’ll just need to set them again and then log out and back in to see if they were really saved. The following instructions are for setting up zap2xml to use Zap2it’s listings, which I believe are slightly more accurate than TV Guide’s for non-prime time listings on certain channels..

1. Register your free Zap2it.com account

2. Click Change Provider to select a lineup (select country and input zipcode)

3. Optionally select your favorite channels (click the channel stars or click “Find Channels” link) to limit channel output

4. You may need to install the required supporting perl libraries (not needed with the Par-Packed Windows file) – COMMENT: I’d try step 5 first before you go adding any perl libraries, as they may already be present.

5. Run zap2xml with the userEmail and password parameters of your account – COMMENT: Watch the output and see it complains about missing Perl libraries. If so, see step 4. Here is how you might invoke it, assuming that you are running Tvheadend and that your Tvheadend user is “hts”:

./zap2xml.pl -u user@email.com -p yourpassword -o /home/hts/.xmltv/tv_grab_file.xmltv -c cache -8 -F -T -q

(Leave off the -q while testing, because it may suppress error messages that you’ll want to see, and leave off the -8 if you have not set any favorites, since the -8 option tells zap2xml to only output favorite channels, such as those you would see when clicking on the “Starred” button in Zap2it). This assumes that you have created a directory named “cache” (as a subdirectory off the directory where the zap2xml.pl script is saved) and that you have either already created a file /home/hts/.xmltv/tv_grab_file.xmltv (can be a zero byte dummy file to start) and made it world writeable, or else that you are always going to run the zap2xml.pl script as the Tvheadend user, in order to avoid file permissions issues. Of course, you must create the /home/hts/.xmltv directory if it doesn’t already exist. The point is that when you run the script and it tries to create the /home/hts/.xmltv/tv_grab_file.xmltv file, you don’t want it to fail due to file permissions or ownership issues. You can read about the options shown above, and others you may wish to use, on the zap2xml home page (in particular, you may want to use something like -d 12 to get 12 days of listings instead of the default 7). NOTE: Should you happen to be running OpenElec or LibreElec, which I most emphatically DO NOT RECOMMEND, zap2xml.pl will not work for you, in part because you may find it difficult or impossible to get to a command line, but also because Perl is not available. There is an alternative for OpenElec/LibreElec that is written in Python, but I have absolutely no experience with it.

6. Optionally set up a cron job/task scheduler task to run it every day – COMMENT: Keep in mind that when setting up a cron job, you must use full paths, and not any shortcuts such as “.” or “~” in the path. If you are using the older method that uses tv-grab-file, the cron job or task should be scheduled to run a few minutes BEFORE Tvheadend’s Internal Grabber is scheduled to run. So if Tvheadend’s grabber is set to run at 2:43 A.M., you may want to run your cron job that invokes zap2xml at 2:28 A.M., giving it 15 minutes to finish (which is, generally speaking, more than ample time for it to run to completion). If you are using the newer socket method, then your cron job should run a bash or shell script that first runs zap2xml, then the appropriate cat command to send the tv_grab_file.xmltv file to Tvheadend’s socket, as described in the first part of this article. Here is a very simplified example of such a script, which assumes that you (as the hts user — use sudo su hts to temporarily become the hts user) have created a directory named /home/hts/listings and placed the zap2xml.pl file in that directory:

#!/bin/sh
cd /home/hts/listings
/home/hts/listings/zap2xml.pl -u user@email.com -p password -o /home/hts/.xmltv/tv_grab_file.xmltv -c cache -8 -F -T -q
cat /home/hts/.xmltv/tv_grab_file.xmltv | nc -w 5 -U /home/hts/.hts/tvheadend/epggrab/xmltv.sock

You would set up your cron job to run this script as the hts user (assuming that “hts” is the Tvheadend username on your system). If you’d prefer to run it in your user directory rather than the hts directory, and to run the cron job as your user rather than hts, you can do that by adjusting the paths in the script accordingly but then you may have to deal with a permissions/ownership issue when attempting to cat the file to the socket.

Whichever method you use, please configure your cron job or scheduler task to run at some odd random time (in other words, not right on an hour, half hour, or quarter hour mark, and don’t use the example times mentioned above) so that everyone isn’t clogging up the servers at once.
(End of edit.)

Particularly if you are running a PVR backend system and can receive channels from multiple satellites, or you live in an area where you can receive stations from multiple TV markets on your terrestrial TV antenna, you may need to set up multiple accounts to get listings from multiple TV markets. In that case, you will probably want to save all the listings from each market area into a separate, differently-named .xml file, and then when you have obtained them all, combine them into your main tv_grab_file.xmltv file. There are utilities and scripts that can help you combine XMLTV format files – see A few Linux utilities that are useful for manipulating XMLTV schedule files. Or if you are using the newer socket method, you probably could just cat each of the individual xml files into Tvheadend’s socket as they are downloaded.

I am painfully aware that there is nothing at all that is easy about this process, and it probably makes you wish that we had European-style free-to-air services, where EIT guide data is embedded right in the program stream (lucky Europeans!¹). But since we don’t, I just wanted you to be aware that you don’t need to have a blank EPG in your Kodi Live TV section. There is definitely a learning curve to getting it all working, but the more you work with it the more you will understand how all the pieces fit together. You may never be able to get schedule data for every channel you can receive, and you’ll obviously never find it for “wild feeds” that come and go, but I’ve been able to populate the EPG grid for quite a few of the stations I’m able to receive on my dishes.

(By the way, if you want icons for the channels, at least in Kodi you have to load those to the frontend system – as far as I can tell, there’s no good way to put them in the Tvheadend backend server and have Kodi get them from there. So just create a directory, dump your channel icons there, rename them to EXACTLY match the channel names except for the extension – for example, if you have a channel named “Big Fart Channel”² then the icon file name should be “Big Fart Channel.png”, or whatever the correct extension is – then in Kodi go to System | Live TV | Menu/OSD and modify the “Folder with channel icons” setting to point to the directory containing the icons. And yes, I’m aware that you can supposedly enter a path to a “User Icon” for each channel in TVheadend’s Configuration | Channel / EPG | Channels tab, but those are really intended to be the paths to “picons” supplied by TV channels in some other parts of the world. I’ve found that if you attempt to use those, more than likely you are going to slow down Kodi or cause some other undesirable effect, such as Kodi hanging when you attempt to quit Kodi. You are better off to store your channel icons on each of your devices that run Kodi.)

If you know of any software tools that would make this easier, or pick up any hints or tips that I have not mentioned, please feel free to post a comment (but be aware that I will not approve spam comments that promote commercial services. Also, I don’t need any praise – if you like this article, don’t tell me, tell your friends that have satellite dishes and that could possibly benefit from this information!).

Here are some possibly useful links:

zap2xml software and documentation.
tv-grab-file, a file used with Tvheadend to import xmltv format data
Zap2xml for ATSC in OpenELEC (Kodi forum thread)
Installing zap2it grabber in OpenELEC (YouTube Video)
Setup instructions and files for Synology NAS users (Tvheadend forum thread)
NextPVR – EPG Setup – XML/XMLTV EPG – Zap2it & Zap2xml (NextPVR forum thread)
Home DVR Tvheadend OTA EPG Setup (Part 2) (YouTube Video)

Note regarding the video in the previous link: It shows the basic setup, but using an older version of Tvheadend, and using the free version of the mc2xml listings grabber which stopped working in July, 2015. So, don’t use mc2xml, use zap2xml instead. Carefully read and follow the instructions at the top of that page on how to set up your Zap2it account, and also consider utilizing the tricks I mentioned earlier in this article.

If you found this article useful, you may like my followup article, A few Linux utilities that are useful for manipulating XMLTV schedule files.

NOTES:
¹ Sure, the lucky Europeans get EIT guide data with their free-to-air channels, but at least we don’t have to pay a “telly tax” on each TV set we own, so there’s that!
² Someone REALLY should start the “Big Fart Channel” – that would be a real gas! And with that, this article has really bottomed out. What do you mean, my puns stink?