Building the Perfect HTPC / Media Server
This post is a break from the normal topic of programming, but has the same level of geekery/technobabble that you have come to expect from my blog. Over the past few years, media usage and needs have been changing/evolving to the point where some people almost completely rid of physical CDs, DVDs, even books. There is also a "cut the cord" movement that emphasizes cutting out any services that you don't make full use of (e.g. cable TV), and favor anything which makes media consumption more convenient.
Here is a quick description of the perfect HTPC / Media Server: Something that can automatically download, repair, extract, gather information on, rename, sort, and play back movies, tv shows, albums, ebooks, games, and even comic books - while making all of the content remotely accessible via browser, mobile-devices, etc. Oh yeah, and it should be able to run 24/7 while using less energy than a (incandescent) light bulb.
While there are a million and a half ways to setup an HTPC (Home Theater PC) / Media Server, I'm pretty satisfied with my current setup, so I thought I would share. In a nutshell, I built a mini-ITX system (for less than $400, which runs at less than 50W) running a combination of Sabnzbd, Sick Beard, Couchpotato, Subsonic, VNC, XBMC, Maximus Arcade and a few other small utilities. With an exception to Maximus Arcade, all of this can be setup using any hardware/OS combination (Windows, OSX, Linux) that supports Java and Python.
Defining Needs / Requirements
Since HTPC / Media Server is slightly vague, here are the specific features I wanted:
Automated Downloading of Movies / TV - Specifically, a way to simulate the functionality of a DVR, without any need for cable TV.
Remote Accessibility - Being able to access/share my media library from the road, work, cellphone, wherever - and only using VNC as a last resort.
Low Power Usage - I wanted to build a machine that could be left on 24/7, without concerns about electric bills or overheating.
Easy to add media, or send links for downloading - Sometimes you think of a show/movie/album you want when away from your computer (at a party, on the road, dinner conversation, etc.) Having an easy way to add items you want remotely is a very attractive feature.
Ability to Play Emulators and ROMs - For me, no HTPC would be complete without the ability to play old console (NES, Genesis, PSX, etc.) games.
From Cable & DVR, to Usenet & Sabnzbd
It's funny how technology can change behavior over time. Back when I had cable, and before I had a DVR, there were only a few ways to catch a movie/tv show - either plan on being home to watch the show/movie when it aired on TV, or rent a DVD / stream from Netflix (assuming the video you want is available, and not too recent). Then, after getting a DVR a few years ago, planning my schedule around when a show came on was out the window. It was a liberating feeling, and let me catch up on shows when I had the time to do so. Not to mention, being able to fast-forward through commercials was a bonus.
After a while though, the novelty of a DVR wore off, especially as the rate and taxes for Cable TV continued to rise. Not just that, but if you ever look into subscriber fees associated with cable, you'll notice that the rates for the various channels is far from equal (e.g. ESPN costs around $4/month, while the Science Channel costs only $0.06/month). So unless you are a heavy sports fanatic, chances are over half of your cable bill is going to channels you'll never watch. I definitely fall into that category, and the linked Consumerist article played a huge factor in motivating me to look for other solutions.
Enter Newsgroups. Most netizens are familiar with direct downloading (e.g. forums, ftp sites, rapidshare, filesonic, hotfile, etc.) or p2p (limewire, torrents, etc.). In fact, just about anyone who has downloaded a digital movie or album online outside of iTunes, has probably done so via direct downloads or p2p. While those are easy and popular options, the reliability and quality of the files, and most importantly - security of the process, are all very questionable. Download sites keep track of IPs, and are riddled with misleading links and malware scripts. Torrents have slightly better quality control, but they are also probably the biggest source of copyright infringement lawsuits. This is because by design, when you download a torrent you are also uploading that torrent to the rest of the world (making you a source for potentially pirated media).
While newsgroups don't completely solve all of those issues, they get close and have other advantages. Most people don't even realize they already have newsgroup access with their current internet provider. Of course, the free access provided by ISPs is very limited, and not anonymous/secure, so this is where a premium usenet provider comes into play.
Premium newsgroup access is typically secure and much more "anonymous" than any other method of downloading (depending on who you get newsgroup access through, and your router/network setup). The other big advantage is that most usenet providers allow you to download as fast as your connection will allow (as opposed to torrents, which are limited to the speed of your peers).
There is an inherent learning curve when dealing with newsgroups, and user-friendly clients have only started becoming more and more refined over the past few years. At some point, a friend pointed me towards an automated usenet downloading solution called Sabnzbd. This actually takes lots of the "manual labor" out of usenet.
As you might have assumed from the name, "premium" newsgroup access is not free. It ranges from less than $10 per month to over $30. For the best of the best, I would suggest Giganews, which ranges from $5 to $35 per month. Second to that, is Supernews, which is run by the same company. Supernews offers a single unlimited service, for a flat rate of $12 per month. There are lots of other options, such as UsenetServer and Astraweb. Both have good reviews and are perfectly decent alternatives.
So what is the actual cost of all this? Well the average digital cable TV bill is around $75 per month. Add internet access, premium channels and a DVR, and the cost easily rises to over $100. That's not even taking into consideration subscriptions to other services like Netflix and Hulu. Similarly, the average high-speed internet connection bill is around $75 per month. By adding another $10 for usenet access, you can have so much more, for so much less. $85 per month is about the top-end of what you would pay for an unlimited selection of tv shows, movies, music, games, etc. There are some other small costs here and there too, such as an NZBMatrix subscription for $12 (for 5 years).
Introduction to Sabnzbd
Sabnzbd (which stands for SAB nzb daemon) is basically a python service which runs in the background, and makes monitoring, downloading and uploading files extremely easy. The entire app runs as a webservice, and is accessible via the browser (e.g. "http://localhost:8080"). Just like a .torrent file is simply an indexed list of links, newsgroup binaries have indexed links saved as .nzb files. Normally when using an NZB file, your newsgroup client (e.g. Newsbin Pro, Unison, etc.) will find all of the "parts" in the NZB, and then download them. Depending on the newsgroup client features, the program may also assemble file parts, repair them (using .par2 files or recovery records), extract them (unzip, unrar, etc.) and even delete the original files (the leftover .rar, .zip, etc.) Truth is, since using Sabnzbd, I don't have a need for usenet clients anymore. Sabnzbd will do all of these things (assemble, repair, extract, delete) and much more (such as renaming, sorting, etc.)
Getting Sabnzbd to download stuff...
So you have sabnzbd installed and setup. How do you get it to actually download something? That's actually the easy part! There are three main ways to send files for sabnzbd to download. First, you can just click the "add nzb" button at the top of the screen. From there you simply upload an .nzb file directly, and sab does the rest. The second way of sending nzb files to sab is via a "watched folder." This is a folder somewhere on your system (e.g. "C:\downloads\nzb") that sabnzbd will constantly monitor for files. As soon as it finds an .nzb file, it will add it to the list of items to download. The third method is to setup an RSS feed through a site like NZBMatrix. More on this in a later tutorial. Those are the basic ways to get sabnzbd downloading... but that's just the tip of the iceburg. The real power of sabnzbd comes into play when you pair it with other services.
Introduction to Sick Beard
As aforementioned, once you have sabnzbd running, you are pretty much good to go. The caveat is that you will need to manually find NZB files to send sabnzbd, which is pretty inconvenient unless you already know what you want to download. Sick Beard is basically the Usenet/Sabnzbd counterpart to a DVR, and like sabnzbd, it runs on Python, and comes with a web interface you access via browser (http://localhost:8081). With Sick Beard, you simply enter a list of shows you would like to watch (either currently airing, or off the air) and it will automatically search for new episodes and send them to sabnzbd when found. As you can see in the screenshot above, The series "FlashForward" and "Testees" are both ended (series was cancelled), however Sick Beard was able to find all of the episodes. Likewise for "Through the Wormhole," which is still airing - all 17 episodes were found and downloaded (aka sent to sabnzbd to be downloaded). You can also see that "Quality" can be set to either SD (standard), HD (720p, 1080p, etc.), Custom (I set custom to look for either SD or HD, and grab whatever it finds), or All (downloads anything it finds, regardless of quality). There is also a convenient field for "Next Ep" which displays the next date an episode is expected to air.
Adding Shows to be downloaded
With a DVR, adding shows is fairly easy. You can add a show by going into a menu and searching by name, or usually by just browsing the DVR Guide and clicking on the show. With Sick Beard, it's just as easy. You can either enter shows by searching (Sick Beard uses theTVDB.com for results) by name, or by searching through folders and extracting the shows from file names of already-downloaded episodes. This is a great feature for anyone that already has a library of tv shows that were found before using Sick Beard.
The Show page, and downloading specific episodes
Once a show is added, you can view the list of episodes and edit any show settings on the page that is created. Here is an example for the TV show "Louie"
As you can see, all of the settings along with some meta-data (grabbed from theTVDB.com) are displayed here, along with an "Edit" button to change the download folder, show name, quality, etc. There are also quick links to each season in the top right, along with specials. Below all of this is the list of actual episodes. Here is a cropped screenshot of the Season 2 list:
Again - Sick Beard takes the list of shows you have entered, and searches for any episodes marked as "unaired" or "wanted". When the NZB file for an episode is found, it sends it to Sabnzbd and updates the status to "snatched." Once Sabnzbd successfully repairs, assembles, and extracts the file, Sick Beard then gets a message to update the status again to "Downloaded." The search button on the right of each episode is a quick way to download a specific one. We'll move on now, since this isn't a Sick Beard tutorial, just a quick overview to give you an idea of what it does.
Introduction to CouchPotato
No DVR is complete if it can only download TV shows - there are also movies, documentaries, straight-to-video releases, and more. The counterpart to Sick Beard (which keeps track of TV shows) is CouchPotato (which keeps track of movies/documentaries). As with both sabnzbd and Sick Beard, CouchPotato is a python script which comes with a web-interface you can access via browser.
Adding movies to the list is very easy, along with selecting the preferred quality. CouchPotato uses a few sources including IMDB and theMovieDB for meta-data like release date, descriptions, ratings, poster, etc. It checks for new releases every 12 hours (configurable) and when an NZB file is found, it gets sent off to sabnzbd for the heavy lifting.
Introduction to Headphones
Headphones has a very similar layout to CouchPotato, and serves the same purpose, but for music. You can enter the name or an artist, or a particular album you want - and it will periodically search for an NZB, and send it to sabnzbd if found. Headphones is fairly new, and development is still in progress. There are a few features that are not fully implemented, and the entire process / configuration isn't the easiest to figure out without a little bit of tweaking. Also, moreso than video libraries, music libraries are notoriously difficult to organize, since there are so many variations of albums, and sources for music. Not to mention the tagging and file naming conventions are all over the place. For those reasons, Headphones freaks out a little bit when you try to import an existing library. I assume that with time, this will become more polished.
Other Automated Downloading Solutions
So far all of these solutions have been centered around sabnzbd (and usenet) being the sole provider of media. There are of course many other ways to automate the searching/downloading process. I won't go into too much detail down these other routes, but I would highly recommend:
For downloading torrents, uTorrent (as of version 3.0) supports remote access, RSS feeds and a long list of other options. It can also be setup to use a web interface, just like sabnzbd, Sick Beard, and couchpotato.
For downloading directly (web links, ftp, file-hosting sites like rapidshare, hotfile, etc.) JDownloader is one of the best apps out there. It supports all sorts of cool features like scheduling, remote control, auto extraction, etc. It supports premium accounts (rapidshare, hotfile, etc.) if you have them, and if not it will take care of the waiting process, redirections, and even has an anti-captcha module which will try to automatically answer any captcha challenges.
The Home Theater part of HTPC
With all of the media downloaded, the next step is to actually play the stuff! For watching tv/movies, the two most popular cross-OS solutions are Boxee and XBMC. Plesk is OSX only, and Windows Media Center is for PC only.
I have used Boxee in the past, and actually prefer it as an out-of-the-box solution. Along with having a standalone unit,
Boxee is also being built directly into TVs now - nevermind, it seems as though that has been abandoned. In addition to playing back video and audio files, Boxee has built-in support for Hulu, Netflix, Pandora, and all sorts of other video sources (youtube, reddit, etc.) There are plenty of skins and addons to explore with the least amount of setup. The only reason I stopped using Boxee was due to some technical issues with video playback and hardware acceleration, along with a personal dislike of the way Boxee handles/buffers streaming media.
My preferred media app is XBMC (which stands for XBox Media Center, since it was originally designed to run on the original Microsoft XBox). Like Boxee, it runs on Windows, OSX and Linux, and is highly extendable. Sick Beard also supports updating of the XBMC library whenever a new show is downloaded. A small feature, but very nice to have. There is less support for other media sources (e.g. Hulu, Netflix, etc.) without third party extensions (which don't work as well as Boxee's implementation). I currently use XBMC because it does not have the technical issues I was experiencing with Boxee, and I prefer the additional levels of customization.
XBMC and Boxee both have "Launcher" apps that can launch a game from within the interface, but they all seem a bit cumbersome. Personally, I would rather use a gaming pc for playing modern games, and if playing games on an HTPC at all, would be via emulators and roms, or perhaps a service like OnLive. There are even specialized extensions for Boxee and XBMC to launch roms/emulators, but again they are a hassle to use. This is where Maximus Arcade comes into play.
Introduction to Maximus Arcade
Maximus Arcade is a Windows-only program and doesn't actually play ROMs by itself. It's a front end for all sorts of emulators for systems like NES, Genesis, Playstation, SNES, MAME, Gameboy, N64, Gamecube, ScummVM, etc. Anyone that plays a decent amount of emulators knows that there is usually a different program for each system, and sometimes even specific settings/programs for certain roms. This is where Maximus really shines. You can configure which emulators launch each type of ROM, and even fine tune the settings for each. Maximus also hooks into EmuMovies to automatically download box art, instructions, descriptions, screenshots and even video previews for your ROMs. Navigating is extremely simple, as you can see below:
It can take quite a while to configure Maximus Arcade (it took me an entire weekend) - since you need to download each of the emulators manually, then configure them one by one in the settings. Once it's all setup though, this program is a dream to use. It's very easy to navigate between ROMs and systems, and the level of customization is almost limitless. Unlike the rest of the apps mentioned so far, this is not freeware or open-source. It's a windows-only program that has a trial download, or a paid version for around $30.
How About the Media Server?
Great, you have a machine that can download and play media exactly how you want. As long as you are in the same room as the computer, that is. Making all of this accessible remotely (from work, on vacation, available to friends, from mobile devices, etc.) is the next step towards perfection of the ultimate HTPC/Media Server. First, let's cover the obvious: streaming audio and video. As long as you setup your home network (router, workgroups, "homegroups", etc.) correctly, you should already be able to access the files on your HTPC from other computers. Both XBMC and Boxee will allow you to stream media from the computer, so there is no further setup required for that. However, this only works within your home network (LAN). If you want to watch a movie or listen to music from work or on the road, unless you have a ridiculously fast connection, there's no real way to stream.
Subsonic is a great little app (windows, osx, linux) that not only handles your audio library, but will also stream video to your browser, tablet, phone, etc. It was originally made to stream audio files (mp3, mp4, flac, wav, etc.) but has since evolved to handling just about any audio or media format. Want to stream a 1080p MKV video to your cellphone/tablet, but your connection is only 3G? No problem, just set quality to low (e.g. 150kbps) and hit play. Of course, if you aren't limited by connection speed, the only limit is how much your HTPC/Media Server computer can handle. Personally, my mini-ITX AMD Zacate that runs on less than 50W can't handle realtime video transcoding, so if that's what you want, an Intel i3 is probably the lowest you can go. For streaming audio however, a low-power CPU is just fine. Making media accessible to friends and family is easy with Subsonic too. You can create accounts for different people, see what media they are playing, and chat with them on the web-based interface. Recent updates to Subsonic have also enabled sharing of songs/albums with direct links, and Last.fm audio scrobbling.
Serving Other types of Media (eBooks, Comics, PDF, etc.)
We could stop here, and most people would. However, this is still an incomplete solution, since digital media also covers written material! For anyone with a tablet especially, this is a great way to have an all-in-one solution. One of the undisputed tools for organizing an eBook collection is the Calibre eBook Manager. Not only does it organize and allow you to read/convert/export your library of epub/pdf/txt/mobi files, but it also has a built-in web-based server!
Outstanding, so now there is a way to access your eBook files, along with all sorts of meta information. Using this in conjunction with a free service like MagicScroll will allow you to not just access, but read your books online. Speaking of reading online, how about a similar web-based service for reading through a comic collection?
Comic Book Archive Server (CBR Server) is a great solution for making your comic collection accessible online. It requires about 15 minutes of setup, since you need to manually download 7-zip, ImageMagick and Java. There are also a few issues that need to be worked out (out of memory errors, missing features, etc.) as development continues.
Accessing and Controlling the HTPC Remotely
With all of these web-interfaces for accessing media (Subsonic, Calibre, CBR Server, etc.) you may still need more ways to get to other files and folders. This is where the following utilities can help out tremendously:
TightVNC is like any other VNC software - it allows you to remotely connect to your HTPC/Media Server from another computer, phone, browser, etc. This is helpful for all sorts of reasons, so highly recommended. There are other flavors of VNC out there as well like RealVNC, UltraVNC, etc. I just prefer TightVNC as I've been using it the longest.
Dropbox is one of my "must-have" tools. There are only two kinds of people... those that use Dropbox, and those that haven't heard of it (or don't understand it). How does it fit into this whole HTPC/Media Server setup? Imagine for a moment you want an easier way to download .nzb or .torrent files. Simply point your sabnzbd/torrent client to a folder inside your dropbox called "dropbox/nzbs" (or "dropbox/torrents"). Now anywhere you have Dropbox installed, you can just save a link and know that your HTPC will start downloading right away - no matter where you are.
Just about all of this article so far has been hardware agnostic. No matter what your setup, you should be able to install all of the applications mentioned so far (except for maybe Maximus Arcade, which is only for Windows). That being said, I mentioned that my personal goal was to have a mini-itx machine that could run on less than 50W, and one that costs less than $400. Since I originally put my machine together, there are probably a few more options that have opened up. Anyways, here is my build list ($426 total as of 9/4/2011, at the time I purchased there were a few sales, so it was under $400):
- $110 - mobo/cpu: ASRock E350M1/USB3
- $60 - case: Silverstone ML03B
- $27 - ram: Kingston 4GB (2 x 2GB) low profile
- $95 - ssd: Intel 320 40GB
- $65 - hdd: Seagate Barracuda Green 1.5TB 5900RPM
- $46 - psu: SeaSonic SS-350SFE 350W (was actually too small for the case!)
- $20 - fans: 2x ENERMAX UC-8EB 80mm Case Fan
For the OS, I installed Windows 7 x64, and did not calculate that into the cost. You may notice I only have 1.5TB of storage, which may seem small for an HTPC/Media Server. Personally, I delete movies/tv shows after I am done watching them - but if I wanted more space, there is enough room to add another two 3.5in drives, so adding another 4TB of space would be trivial. I also purchased a compact power supply unit (PSU) thinking that the case needed it, however the Silverstone case was actually made for a regular size PSU. Not really a big issue for me, since I was stuffing the unit into an entertainment unit. It runs 24/7 inside of a small space in a wooden entertainment unit, uses around 45W, and the temperature hasn't gotten over 48C.
Nothing is more annoying, or more certain, than having to buy a few extra odds and ends to complete an HTPC setup. Your machine may have the perfect combination of hardware and software, but there is always room for improvement. Here are a few optional additions that can make the difference between a headache and a home theater.
Slow LAN / Buffering Issues / Streaming over WiFi
After my setup was complete, I had intermittent issues with buffering. My network speeds were good, and the WiFi signal was decent, but not always perfect. Turns out, the actual placement of my wireless router/modem, and the placement of the WiFi antenna on my HTPC had too many obstructions between them (walls, beams, appliances, etc.). I almost considered running a 50ft ethernet cable across the floor (or in the ceiling) until a friend pointed me towards Powerline Ethernet, which saved the day. Specifically, the NETGEAR Powerline AV 200 Adapter Kit ($100).
The Powerline kit basically lets you use your electrical system as an ethernet cable. You plug one of the adapters into a power outlet next to your modem/router, and then connect it to the modem/router with an ethernet cable. Then, anywhere else in the house you plug the other adapter into an outlet, and run an ethernet cable from the adapter into your HTPC. Instantly, all of the buffering issues were gone. There aren't many products out there which are truly "plug and play," but this is one of them. No setup needed whatsoever, it just works... and after troubleshooting network issues for over a week, it was $100 well spent.
HDTV Losing Connection with HDMI (Black Screen)
This was an extremely annoying problem. Sometimes, when I turned the TV off, or the HTPC went to sleep - the connection between the HDMI on my computer and the TV would get lost. I would have to manually turn off the HTPC, sometimes I even needed to unplug the power and the HDMI cable before I could get the TV to communicate again. Turns out this is actually a fairly common problem with HTPCs and certain types of TVs. What's happening is that the TV has an EDID (Electronic Device ID) which the computer uses to identify where to send the signal. There is some kind of a "handshake" going on during this process - and if at any time the communication between HTPC and TV is broken, it can be a hassle to get the signal back. This is where the Gefen HDMI Detective Plus shows it's worth. It's painful to drop another $85 on a "budget" HTPC, especially when it's close to 1/4 the cost of the whole machine. Then again, it's hard to put a price on peace of mind. After reading the Amazon reviews for this unit, it wasn't a very hard sell.
Mouse, Keyboard, Remote
There are lots of options for HTPC remotes out there. Some people swear by Logitech Harmony remotes - some like Boxee remote, etc. I already had a Windows Media Center USB remote and love it. The backlit rubber buttons are great, and the USB receiver seems to work from anywhere in the room (which is more than I can say for a handful of other remotes I've tried). I believe it works on RF, so it doesn't need a line-of-sight like InfraRed. On Amazon the remote is pretty expensive, but I'm guessing that's because it's been discontinued in favor of a newer model.
As for keyboard/mouse, I almost went for a Logitech diNovo, but was scared away by the price and bad reviews. Instead, I opted for a Logitech unifying Couch Mouse and a cheap unifying wireless keyboard. They don't get used much, unless I'm using the HTPC as an actual computer (or for internet browsing / emailing / etc.). When I do need to use them though, they are invaluable - since typing something out with a remote can take about 50x longer than if you have a regular keyboard close by.
So what's the real cost?
If you have been keeping track, this budget, sub-$400 HTPC has slowly gotten more and more expensive with each of these additions. With the Powerline Ethernet adapter, HDMI detective, and remote/keyboard/mouse, the final cost will probably be closer to $600. When I first set out to make an HTPC, I kept reading over and over to take my intended budget and double it. Turns out, that was correct almost to the dollar (my original budget was about $350).
But Wait, That's Not All!
There are also lots of browser extensions and user-scripts made for Sabnzbd, CouchPotato, uTorrent, JDownloader, etc. I may add some updates to this article in the future to cover those as well. For now, this should be more than enough to inspire you to "cut the cord!"