Could satellite delivered data streams end the tyranny of usage caps imposed by many North American ISPs?

Ku-band satellite dish

A Ku-band satellite dish – By Ranjithsiji (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0) or GFDL (http://www.gnu.org/copyleft/fdl.html)], via Wikimedia Commons

It’s not a big secret that one of the most common uses of the Internet is viewing video. And while many people are watching streaming video these days, a lot of the video that is being viewed is low quality, low definition video due to the internet usage caps imposed by many Internet Service Providers (ISPs). Not everyone is lucky enough to live where Google Fiber, or some other high-bandwidth, usage-cap-free ISP offers service. When there are overage charges for bandwidth usage, that discourages customers from using the Internet to view video, or at least from viewing it with the highest possible quality. Most Internet users would like to see usage caps eliminated altogether, but the ISPs that have them see them as a potential source of additional income.

If a significant percentage of the video that users watch could be shifted to some other transmission medium, users would find it much easier to stay within their usage caps, and the ISPs might be persuaded to someday eliminate them altogether. Unfortunately, the only other options available to most people are direct-broadcast satellite (DBS) or cable TV, or over the air transmissions received using an antenna. But the price of satellite and cable TV bundles is increasingly becoming unaffordable for many people, and people in rural areas or smaller cities may have difficulty receiving over the air transmissions, or may not have a wide choice of channels and/or programs to view.

Most Americans do not realize that in Europe there is another type of satellite television, known as Free-to-Air. This should not be confused with what we commonly referred to as “free to air” in North America. The European version of Free-to-Air is similar to what North Americans might receive if they lived in a very large city with many channels available via an antenna. However, it is delivered by satellite, and there are a large number of satellite channels available. Some require payment, and use conditional access cards to authorize access, but many other channels are truly free to view. While we might wish that this type of satellite service were available in North America, it doesn’t appear that we’re going to get anything similar in the foreseeable future.

In contrast the North American version of “free to air” consists mostly of audio and video signals sent via satellite, that are intended for reception by entities other than home satellite viewers. Typically, these signals are intended for reception by commercial entities such as television stations and cable systems, which then rebroadcast them to their viewers or customers (often with degraded quality). They are called “free to air” because they are unscrambled (not encrypted in any way) and therefore can be received by anyone with the proper equipment, but (with rare exceptions, usually religious channels) they are not primarily intended for home viewers.

There are many problems with the North American version of “free to air” such as the fact that signals are spread all across the satellite arc, and therefore require a movable dish or several fixed dishes to receive. Worse yet, what many people consider to be the best content is only available on C-band, which can only be received by the larger dishes of the type you used to see on farms and in rural areas around the country, but which are not permitted nowadays in most cities. And installing a dish takes a certain amount of technical knowledge that most people do not possess (this knowledge can be acquired, but there is a definite learning curve).

Another problem with “free to air” in North America is that for the vast majority of available channels, no schedule information transmitted as part of the satellite feed, making it difficult (and sometimes impossible) to reliably schedule upcoming programs. Whereas, the European version of Free-to-Air and the North American commercial DBS providers do send schedule information to receivers.

The reason that DBS services like Dish Network, DirecTV, Bell Expressvu, and Shaw caught on in the first place was because they use a single small dish that is much easier to install and aim. Plus, they are capable of receiving multiple signals at once, so you don’t have to forgo recording one program while you are watching another. This is the same type of dish used for the European-style Free-to-Air services. And it goes without saying that the commercial services offer a much wider variety of programming, but the problem in North America is that you cannot choose to pay for only the type of programming you want, without having to pay for types of programming you don’t want. For example, most people know that a large portion of the commercial cable or satellite TV service bill is due to the inclusion of sports programming, but if you never watch sports, you’re paying a significant sum each month for programming you will never watch.

In contrast, Internet-delivered services are starting to offer package deals where you pay a flat monthly fee, and can then select what you want to watch from a list of choices. Some of these services are better deals than others, but the problem remains that if you watch much video via these services you run the risk of blowing through your bandwidth caps.

What if you could have the best of both worlds?

Many companies perceive that satellite time is very expensive, but that is because they have only used it for real-time video. The companies that own satellites have different ways of charging for services and while I do not know all the details, I have heard is that it is much cheaper to send narrowband data transmissions as opposed to something like full high definition video. And what is video sent over the Internet? It’s simply data. When you send a video clip of a child’s birthday party to a grandparent far away, you’re just sending a data file. It’s not in real-time – the file may take much longer to send than the actual time of the recording – but the recipient doesn’t care if it took ten minutes to send that one minute video, as long as they can view the recording once it’s received.

There is a lot of unused time available on commercial satellites. Anyone who has a receiver capable of receiving the North American version of “free to air” knows that there are times when a large percentage of a satellite’s capacity is not being utilized for anything useful. What if all that currently unused capacity was used for sending data streams? Video content could be encoded and placed into those data streams, and consumers could buy receivers or PC tuners that would receive the data and convert it into video. Both free and paid content could be distributed in this matter. The free content could simply be converted into standard video files. The paid content would only be saved and decoded if the customer had paid for it. All types of content could be sent this way, everything from old-style standard definition programs through the latest 4K or 8K content. The larger the file, the longer it would take to transmit, but since the customer would not be watching it in real-time anyway, he or she wouldn’t care.

Let’s take an example of how this might work. Suppose you have a one hour program that is supposed to be sent out every week and made available for viewing at 8 PM on Monday night. Sometime in the previous week, not necessarily at any scheduled time, your program would be placed on a satellite data stream and sent to all authorized receivers. The program could be flagged to not be made viewable until the appointed time. It may take hours to actually transmit the data, but the receiver will quietly sit there and gather all the data until it has the entire recording. Again, the viewer doesn’t care as long as the entire program is received, and can be viewed on or after the appointed time.

But, you may wonder, what happens if the entire program is not received? Well, there is a technique that used to be used in the old days of Usenet (anyone remember that?) for sending large files. These files were compressed using the RAR format, but after the main file was sent, a series of .par files were sent, which could be used to recover missing data in the original archive. The magic was that if you had, for example, two missing or corrupted data blocks in the original archive, you only needed two blocks from ANY of the associated .par files to fix the data (see the Wikipedia article on Parchive for more information).

A similar technique could be used with satellite data. After the transmission of the full data file, a number of .par type recovery blocks could be sent, and the receiver could apply these to the original file, if necessary, to fix any missing or corrupted data. But beyond that, those .par files could also be sent over the Internet. So, let’s suppose the receiver got 80% of a program and then the power failed, and therefore it missed the remainder of the program and the repair blocks. When it came back online, it could request enough repair blocks to complete the original file from an Internet source. All of this would be transparent to the end user. This technique could also be used to repair data lost due to short periods of “rain fade”, where intense rain disrupts the signal for several minutes, or signal loss due to “solar outages” (periods when the sun is directly behind a satellite, overpowering its signal) in the spring and fall of the year. In this way, video files could be made 100% complete, and therefore provide even more reliable service than what is currently seen with many commercial providers. This would be even more true if the stream and/or the .par-style recovery blocks were transmitted multiple times, several hours or days apart.

The big advantage of this is that a programmer could choose to send all of its data via a single satellite (or perhaps a pair of them, one in the eastern sky and one in the western sky relative to North America, so that users that could not receive one satellite due to obstructions such as mountains or tall buildings might be able to get a clear signal from the other). There are many Ku-band satellites in orbit around the earth and in most cases their signals can be received relatively reliably using a dish of one meter or less in size (such as the one pictured at the top of this article, or perhaps just a bit larger).

Obviously, this type of programming could not be received on a conventional satellite receiver. But a variation of this technique is apparently already being used to send syndicated programming to television stations. The programs are sent by satellite, but as data. I have no idea if the type of error checking and correction I have mentioned is being used – for all I know, they may have even better techniques for dealing with missing and corrupted data – but the basic idea of sending programs as data is already in use. And yes, some hobbyists have already figured out how to receive these transmissions. Apparently some newer PC tuner cards, coupled with the right software, will receive and download these data streams, and when they are downloaded they can be processed with a program called IPCleaner to recover the video. The big problem for the home or hobbyist viewer is that only the intended recipients know when a particular data stream will be sent, therefore (to the best of my knowledge) there is no way to reliably record every episode of your favorite syndicated program that is sent in this manner.

But if someone would set up such a service that was specifically intended for home viewers, they could make sure that their receivers know how to find the programming requested by the customer at the time it is sent. It seems to me that this type of service offering should appeal to services such as Amazon, Netflix, Redbox, and even Google and Yahoo, and in fact all companies that would love to be able to send high quality video to customers but not have to deal with demands of the large ISP’s and their usage caps (not to mention their occasional threats to degrade the speed of a connection unless the sender pays what amounts to extortion, in order to assure that their customers will be able to stream high quality video).

The only downside of such a service would be that, as with the original Netflix service, customers would need to indicate what they want to see in advance, so the receiver could download the desired data when it is sent. But as the company learns the customer’s preferred program choices, they could suggest new programs to the customer, and even in certain cases instruct the receiver to download premieres of new series similar to those that the customer is currently viewing, and then offer those to customers for immediate viewing. This type of system could be very open and flexible, and utilized in some very creative ways. Or it could be closed, like the original AOL concept of the “walled garden”, in which case I predict most consumers would turn up their noses at it, unless perhaps the content is either very compelling or completely free.

I would love to see someone set up a truly open, customer-centered service of this type – it might even make a good Kickstarter-type project. Your role would be to design the receivers and the protocols that would be used, and then you would offer your services to programmers wishing to get their content to customers. You would, however, need to find a team to design your tuner cards and/or receivers, or perhaps adapt existing satellite tuner cards with conditional access slots (intended for use in Europe and other places that have the type of Free-to-Air service I mentioned near the start of this article, but strangely they are available in North America) for this purpose. There is really no reason to restrict the service to a single platform, and a service that can be utilized with many different types of PC’s and devices would be far more attractive than one restricted to a single type of set-top box. Devices such as Roku boxes may be popular with a certain segment of customers, but they do limit your options.

Perhaps the biggest advantage of such a system would be that it would allow users to select programs on an individual basis.  No one would be forced to pay for programming they have no interest in.  Companies might elect to offer service tiers, where for a flat monthly rate you can download all the programs you can watch or for less money, a maximum of a certain number of hours of viewing per week or per month.  High cost programming, such as major sporting events or newly-released major motion pictures, could be offered only to those customers that actually want it, perhaps at a slight extra cost.  But keep in mind that with this scheme, when you are transmitting the data there is no cost difference in sending the stream to a thousand viewers or to ten million – your costs to uplink the data to the satellite are exactly the same either way.  So, generally speaking, there would be no reason to restrict how many shows a subscriber could watch.

In a way it would be similar to traditional DBS or cable service, where you can watch 24/7 if you want to without increasing your monthly rate, but instead of watching channels you’d be watching programs.  You would not need to buy a bundle of channels that include AMC if the only thing you want to see is Breaking Bad, you’d simply select that one show as one of the shows you want to watch (yes, I know that Breaking Bad is no longer in production, I was just using it as an example of a popular show that for many people was the only show they watched on that network).

In many ways this would be similar to the newly launched CraveTV service in Canada, except that it would not be affiliated with a Cable TV provider, and it would not exactly be “on-demand” due to the necessity of selecting the desired data streams in advance.  But it would provide two advantages that customers really want: Much lower cost than traditional cable or satellite TV service (in fact much programming could be offered free with commercials inserted to defray the costs) and the ability to select only the programs they really want to see without paying for expensive bundles of programs they have no interest in.

I also just want to point out a few other things. First, once you get large enough, there is no reason you could not have a high-bandwidth stream or two to transmit “live” content that loses value if it is time-delayed (such as major sporting events). Nowadays there are compression algorithms that allow the packing of a half-dozen or so real-time high definition streams onto a single satellite channel. Yes, that would be more expensive but if you need it to attract a certain class of customer, it might be worth it. So you don’t want to start there, but you might want to keep that possibility in mind when designing your receiver or software.

Second, for this type of system to work, it should contain that largest hard drive that is reliable and affordable, and there should be USB3 ports that allow customers to add additional external hard drives (also the internal drive should be replaceable by tech-savvy customers). The reason is that high definition programs can consume a LOT of storage space. And it should have Bluetooth connectivity, so that received videos can be transferred to a user’s phone for later viewing.

And finally, if a customer really wants some “specialty” programming that has so little demand that you can’t justify putting it on your satellite stream, there is no reason such one-off programs can’t be sent to the receiver entirely over the Internet. Unless a customer watches nothing else besides those types of programs, they will still limit bandwidth usage by watching most programming from the data sent over the satellite.

I would be remiss if I did not point out one HUGE advantage of this scheme from a marketing standpoint: The operator of such a service would have up-to-date information about which programs are really popular.  No more guessing about how many homes are really watching a particular program – if a user takes the time to select that program, you can be reasonably assured they are watching it.  You could also make comparisons, such as whether 100% free programming with commercials does better than subscription programs with fewer or no commercials.  Obviously, if you do this in anything other than an anonymous manner, people are going to be very upset with you.  No one wants to be individually tracked.  So just because you have certain capabilities doesn’t mean you should use them.  But even the anonymous data you could gather might be worth money to advertisers and marketers.

I’m literally just touching the tip of the iceberg here as far as exploring the ways such a service could be implemented.  But a lot of people would benefit if it were.  Ku-band satellite owners could start putting all that unused satellite time to good use. Programmers of niche and regional programming could have a cost-effective way to get their programs delivered to users.  Users would benefit from the much wider selection of programs, the ability to watch a show at any time once the data stream has been downloaded and saved, and of course the cost savings.  If I sat here and thought about it long enough, I’m sure I could think of a lot of other reasons why such a system could revolutionize content delivery.

And the real beauty is that if and when the greedy ISP’s finally drop their usage caps, you can seamlessly switch a higher percentage of your content to Internet-based delivery, since the receivers would be able to handle content delivered either way.  In effect you are future-proofed – if one delivery method becomes too costly, you simply switch more of your content to the other.

Do you think such a scheme would work?  Or would the fact that potential users need to decide what they want to watch in advance totally kill it, despite the cost savings?  If you have any thoughts on this, please leave a comment!

Advertisements

How to play video recorded from high-bitrate 4:2:2 sources on low-power systems

In a previous post, I told you about Fixing the audio on recorded programs from a certain network (which shall remain nameless). The technique for doing that involved the use of the ffmpeg software, and I suggest you read that article because it gives some information near the end of the article on how to obtain a correct version of ffmpeg. The technique I am about to discuss also uses ffmpeg.

If you are running a PVR/backend system such as TVHeadEnd, you may have tried recording programs from certain high-bitrate 4:2:2 feeds, and then found that they simply would not play back on your home theater PC device, particularly if you are using something low-powered like a single-board computer. You might get sound only, with a black picture (and maybe a few flashes of green or some other color), or the video just might play really slowly for a few seconds and then start breaking up. The solution to this is to use ffmpeg to convert the recorded program after you have finished recording it. This means you can’t watch the program live, but you can view it later.

When I first ran across this problem, I found a nice GUI-based program that could do the conversion, but it also took a ridiculous amount of time (about 8 hours to convert an hour of video). But also, it not only converted the video, but also the audio, which destroyed any hope of hearing surround sound. I then figured out that it was actually using ffmpeng to do that conversion, and that by running ffmpeg with only the necessary options I could cut the processing time down to around one-quarter of the time taken by the GUI-based program (or around twice the original running time of the program), plus I could set it to pass the audio and any subtitle streams unchanged. The line I used looks like this:

ffmpeg -loglevel quiet -y -i “original_recording.ts” -c copy -c:v libx264 -b:v 16711680 -pix_fmt yuv420p “converted_program.ts”

(The above is a single line, even if it does not display as such here.)

The -b:v 16711680 option sets the bitrate for the conversion and is the same as the one used by the GUI-based program. In my case it seems to produce a file that’s somewhere around half the size of the original. The video quality is still excellent. By the way, the -y option tells it to overwrite any existing file with the name you specify for the converted program file; if you want it to prompt you when there’s a file name conflict then leave that option out. The above line assumes you are converting a transport stream (.ts) file, if not, then change the extensions accordingly.

If the resulting file is still a little too “fat” for your equipment, then you can try reducing the -b:v value, with a corresponding loss of quality. The GUI-based program could use -b:v 7325135 and still considered that “high” quality, while -b:v 4229320 was considered somewhere between “high” and “standard” quality. Of course, you could go even lower, particularly if you will be playing the recording back on a much smaller screen. And, I do not think those numbers are in any way magic, and there is no reason you couldn’t stick with even decimal or hexadecimal numbers for your trials, so for example you could use 16777216, 8388608, 4194304, or 2097152 (corresponding to 0x1000000, 0x800000, 0x400000, and 0x200000 in hexadecimal) as your -b:v trial values. You just need to find that sweet spot where the video is of the best possible quality that will play without problems on your equipment.

There are many other options you can use with ffmpeg, including ones that will reduce the screen size in case you are watching on an older computer or tablet, or something else that does not have a 1080p screen, but I just wanted to show a simple conversion using the fewest number of options.

I will note that ffmpeg has a tendency to gobble up as much available CPU time as it can get while converting video. If you fear that it will steal needed CPU cycles from other programs (such as your backend software), you can use nice to keep it in check. For example, if you use:

nice -n 20 ffmpeg […options…]

It will cause ffmpeg to run at the lowest possible priority and give up CPU cycles to any other program that needs them. Nice can take values from -20 to + 20, with the higher the number indication how “nice” the program will be to other programs. Negative values will raise the priority of the program higher than normal, while positive values will give it lower priority than most other programs.

Similarly, you can use ionice to keep ffmpeg from hogging access to your storage devices. If you use:

ionice -c 3 ffmpeg […options…]

That will cause ffmpeg to give preference to any other program that needs access to your storage devices. You might want to use this if ffmpeg will run at the same time your backend software is recording programs to your hard drive. And yes, you can use both ionice and nice together:

ionice -c 3 nice -n 20 ffmpeg […options…]

Note that if you are running this from a shell script then you may need to specify the full paths to all the programs used, like this:

/usr/bin/ionice -c 3 /usr/bin/nice -n 20 /usr/local/bin/ffmpeg […options…]

The paths shown above are valid for most Debian/Ubuntu based systems. To find out where these programs are located on your system, you can use the which command. For example:

$ which ffmpeg
/usr/local/bin/ffmpeg