This is going to be another quick and dirty guide for setting up static NAT aka 1:1 NAT aka 1 to 1 NAT on IPCOP.
Commercial grade routers are expensive beasts. A run of the mill Sonicwall Pro Firewall with DMZ support with set you back well over a thousand dollars. They’re nice but you also have to pay an extra monthly fee if you want fancy features enabled.
So you have an old PC lying around and a couple of spare network cards, or maybe you get Soekris 4801-60 ($260 see photo). And you’re thinking to yourself “Well, IPCOP is a free download, I just saved $1200 dollars!”
Not so fast buddy. IPCOP can do most of what my fancy Sonicwall did but you’re going to have to get your hands dirty.
Good Cop:
Free
Very powerful on a good computer
Intrusion detction using Snort (not intrusion prevention)
DNS caching, transparent proxy, lots of graphs, basic QOS
Basic VPN support
Open Source
Bad Cop:
Not very intuitive, can be frustrating to install
Advanced features require add ons and text file editing
Intrusion detction only detects, you need Guardian to beat up the intruders
DMZ setup is quirky, no transparent DMZ mode
Tech support limited to forums, I used to know the Sonicwall guys on a first name basis
These are the basic steps involved in getting a firewall running including a DMZ for your servers with multiple real IP addresses which will be separate from your green LAN zone. So this is a Red, Orange(DMZ), Green(LAN) setup.
* Set up aliases – This is pretty straight forward, read the manual. You’re basically telling the router about the static IPs associated with your internet connection here (at least those you’re planning on using). Our T1 came with 16 or so.
* Set up port forwarding – Again, just read the manual. Forward port 80 from your virtual IP addresses to your DMZ boxes.
Now you can visit your sites from the outside but if your server connects to an outside site it will appear to be coming from the RED interface on your firewall instead of the alias IP you set up. That’s a problem for a lot of reasons. Email, firewall rules from other machines, etc., start to freak out.
Here’s the fix:
* Turn on ssh (system, SSH status, enable)
* Use putty or whatever, to log into port 222 (not 22). (or just use the router’s keyboard/monitor)
* Edit your /etc/rc.d/rc.firewall.local file. It’s very important to put this in the right place. This should go beneath the line containing start) and above the line containing ;;
/sbin/iptables -t nat -I POSTROUTING -o eth1 -s 192.168.1.2 -j SNAT --to-source 23.23.23.23
Make sure you change eth1 to eth0 or whatever your red NIC is named. (go to status, network status and look for the red font). Also, change 23.23.23.23 to whatever the is external IP you want to use (same IP as used in the alias step above).
Reboot. Test it. On your DMZ server install lynx, the text based browser. For Ubuntu that’s sudo apt-get install lynx
At the command line type lynx -dump whatismyip.com
That should spit out something like Your IP is 23.23.23.23
If your DMZ server is running Windows just visit whatismyip.com in a web browser. It should return the IP address of the alias IP you configured earlier and NOT the IP address of your router’s red interface.
*note IPCOP will not let you simply set your DMZ servers up with real IP addresses and use transparent DMZ mode like you can with the Sonicwalls. You have to put them on a subnet separate from the red interface and port forward.
*note2 – Because IPCOP can’t support transparent DMZ mode you have to set the gateway for your DMZ boxes to the DMZ IP address of your IPCOP and not the one provided by your ISP.
*If you want to turn on intrusion prevention read the following words I found at Snort.org
Just to clarify, guardian is not an addon to ipcop, it’s just a program that read snort files and modifies the linux firewall using iptables. Once you get a good sample of your network traffic viewing snort logs, you should get a general idea of what to enable/disable in the SNORT rules. To test it, just run a port scan to the device, and then try to go into the internet from the same device. To make extra sure the blocking is done, you can vi the iptables file in /etc. You should see the ip that’s blocked. It’s not hard to set it up. What’s harder is to configure rules in snort and the ignore list of guardian.
You can get a sneak peak at the site I’ve been working on by clicking on the logo and entering woofwoof as the password. We’re going to launch publicly in the not too distant future.
I love the irony that is Trader Joes. It’s a shining beacon of capitalism frequented by anti-capitalists. Sometimes, when debating the Left, I like to use their favorite grocer to make a point. The left wants socialized medicine because capitalism shouldn’t be trusted with something so important. When I debate people that don’t know me I like to take it further.
Me: “Yeah, I totally agree. Food is at least as precious to the nations health so we really should get rid of food stamps and socialize food as well. We could set up government run supermarkets and copy the policies and procedures used by the DMV to manage the operations and lines. Then, once we got the hang of it, we could get rid of private doctors.”
At this point they realize my logic is perfectly aligned with theirs but, in the back of their head, they’re imagining waiting in line for a half hour for a bottle of Two Buck Chuck that has been replaced with Schwarzenegger Vineyards. Which would require 40 tax payer dollars and 38 unionized workers to make it to the shelf. But hey, it’s free!
We can give food stamps to the poor and they can shop at Trader Joes so why can’t we do the same with health stamps instead of socialized medicine? The typical response to that question goes something like. “Well, that addresses the poor, but I wouldn’t be caught dead using food stamps.”
So I’ve been getting closer to the crux of the issue. The implication of the left’s stance is that they don’t want free healthcare in the form of stamps because the government is much better able to hide the wealth transfer. People would rather get crappy but “free” healthcare from a bureaucracy than swallow their pride and go knock on their rich neighbor’s door to get better healthcare.
In other words: Hiding the wealth transfer from the rich to the middle class (using a layer of bureaucracy) allows the belief that we can all be productive members of society to continue even when we’re getting handouts. The reason we have food stamps and not health stamps is because the middle class can afford food. I would predict that if the wealth divide got big enough people would shun food stamps and demand that the government take over Trader Joes and let politicians run the supermarkets. Because that’s apparently more dignified.
* Note: I don’t really like the terms upper and lower classes because it implies that the rich are somehow morally superior. Click the photos for their Flickr pages.
There’s a saying about startups. That you have to work 12 hour days. Well it’s all a lie. It’s more like 14 hour days. I’m really, really in the zone right now though. My brain’s hanging in there but my eyes are nearing their limit. Visine is keeping me in the hunt.
I’m presenting the site to a bunch of investors on Monday. I’m going to stand up in front of a executives and talk for 20 minutes. Normally I’d don’t like talking in public but I can ramble on about journalism and technology for hours at a stretch.
Artist is working on a logo for tomorrow. Looking for security holes. Fixing bugs.
Right now HoundWire.com is password protected but if you’re interested just leave me a message at
kirk@YOURPANTSabinventio.com Just be sure to remove YOURPANTS first (that’s a creepy anti spam technique FYI).
Looks like I might have a patent app in the works for an advertising idea I had this afternoon. Some lawyers are looking into it.
New RadioHead album is definitely worth the price, whatever it is. Also listening to M.I.A. Imagine Bjork in a jungle hunting Gwen Stefani down with an AK-47 and you’ll get the gist of it.
I can say, without hesitation, that I’ve learned more in the last few months than during any other point in my life.
Here’s some marketing stuff I’m working on for HoundWire.
We aim to provide a replacement for the newspaper and an outlet for all local writers and journalists. Owning a printing press shouldn’t be a requirement for communicating with other citizens. The front page of any given community consists of news voted on by the users. You can also vote on comments so if someone is consistently insightful it will be apparent.
” Alas, I find the Web 3.0 arguments as clear evidence that the proponents don’t understand Web 2.0 at all. Web 2.0 is not about front end technologies. It’s precisely about back-end, and it’s about meaning and intelligence in the back end.
As Paul Graham pointed out, the cost of starting a web based business is approaching zero. As someone who’s starting up a .com I can say that the cost has definitely not reached zero yet but Mr. G’s point remains correct. The new barrier to entry is knowledge. Servers may be cheap but a lack of knowledge means you’ll need to bring in expensive consultants to setup your server and network. My goal here is to help the person with an idea and some basic coding skills to get a simple but fast server up and running.
I’m going to cut to the chase for those short on time (it’s a 2800 word post). In order of effectiveness, here are the tweaks I made to my setup to increase performance:
Index tables – easy to do, massive speedup
Try switching your tables from MyISAM to InnoDB. This is not a sure thing and there are tradeoffs but it really sped things up for me
Turn on Apache’s mod_deflate. My pages dropped in size from 100KB to roughly 17KB. Faster page loads, less bandwidth.
Tune your my.cnf file. Out of the box MySQL assumes you have a very slow server.
All said and done, without changing the software, I dropped page load times from .4 seconds to about .04 seconds. Making the thing run 10x faster using more hardware would have been rather moronic
If Web 2.0 is referred to as the Read/Write Web then Web 1.0 should be remembered as the Read Web. The implications for what type of server you’ll need are huge. You could get away with slow drives back then because your rarely changing data would probably be cached in RAM. If you look at a site like Digg it’s another story. Hundreds of people are writing comments, submitting stories, voting on stories, and engaging in various other activities that can be logged to improve the site.
(Updated thoughts: If the CPU again becomes the bottleneck then optimization will become arguably more important. You’ll just optimize for the CPU instead of IO. My application has a quality dial. I can scale back the quality of the results depending on the load on the server.)
There are lots of good bits of knowledge about the various steps involved in building a LAMP server scattered throughout the web but nothing I’ve found really compiles the information into a usable guide. This post is going to focus on server hardware and networking. If this post gets a decent response I’ll take my book of notes about software and turn it into a software HOWTO.
Just a little background. I’m not a the best software engineer nor a hardware expert. The site I’m launching, HoundWire.com, is really about journalism and geography (that’s ‘hyperlocal content aggregation’ if you’re a hipster). The fact that I was able to build a prototype without bringing in expensive consultants and expensive hardware probably didn’t hurt my cause when it came to getting funded.
I’m going to assume that you’re building a database driven web site using Linux-Apache-MySQL-PHP. LAMP is a good place to start if you’re not a computer scientist and just want a working prototype.
Hardware:
8 Gigs of RAM now costs in the neighborhood of $300. Dual core processors are fast, cheap, and getting cheaper. Run of the mill PCs are equipped with stupendously fast server grade PCI-Express slots. For under a thousand dollars you can put together a seriously fast system with one major shortcoming. The storage system.
From chapter 6 of the MySQL high performance Book:
The fundamental battle in a database server is usually between the CPU(s) and available disk I/O performance; we’ll discuss memory momentarily. The CPU in an average server is orders of magnitude faster than the hard disks. If you can’t get data to the CPU fast enough, it must sit idle while the disks locate the data and transfer it to main memory…
This all means that the first bottleneck you’re likely to encounter is disk I/O. The disks are clearly the slowest part of the system. Like the CPU’s caches, MySQL’s various buffers and caches use main memory as a cache for data that’s sitting on disk. If your MySQL server has sufficient disk I/O capacity, and MySQL has been configured to use the available memory efficiently, you can better use the CPU’s power.
IOPS are a good measure of disk performance on databases. The average consumer grade drive can handle 100-150 IOPS. One 15,000 RPM Seagate Savvio is in the 300s. My Raptor RAID array can probably handle 400+. Now consider that a good CPU can handle nearly 100,000 IOPS. Super expensive RAM based drives are useful because they eliminate the drive bottleneck.
If your database rarely changes then disk IO is much less of an issue because most of your data will be cached in system memory anyway. But newish websites like Digg or Reddit have constantly updating discussions in their comments sections. If you want to harness the brain power of the masses you’re going to need a setup that can write that information to a disk at some point.
Desktop PCs were rarely used as servers in the past because they were limited by the PCI bus. Gigabit network cards and a RAID array could easily swamp the meager bus. Now we’re blessed with PCI-Express and the difference between a desktop and server has more to do with reliability than performance. Reliability is nice but you can save a ton of money if you don’t need it.
The bottleneck on your fledgling system probably isn’t going to be your quad core CPU. If you’re squeezing Apache and MySQL into the same box to keep costs low you’ll need to have a good storage setup. So that’s where I’ll begin.
Hard drives are like sports cars. If you take a 4000 horsepower funny car to the Nurburgring it’ll probably lap slower than a Mazda Miata. A drive that boots Vista in 9 seconds may not be very good at randomly writing to a database, which is what you’re probably going to be doing. Drives that specialize in high load database activity, like the much heralded Seagate Savvio, can be had for around $350. That will only get you 36 Gigabytes but you have to ask yourself; is your web app really going to need a Terrabyte of storage? You can argue that more RAM will solve any problem but at some point user input has to be written to the database.
Most consumer grade PCs do not have SAS connectors but you can get a PCI-Express SAS adapter for under $200. So for under $600 you can turn your funky desktop PC with a PCI Express slot into a pretty darned powerful database server. You can add a drive and turn it into a RAID 0 array (72GB) for around a grand. SAS drives are designed for enterprise use and so are much more reliable than a re-purposed desktop drives under heavy load. In other words your RAID 0 array will last a longer.
I went the cheap route and put two WD Raptors (SATA instead of SAS) in a RAID 0 array using a 3Ware RAID card. Ill know within a few months whether or not my idea is going to take off so I’m more concerned with speed and cost than reliability. Whatever you do, don’t plug a SAS drive into a SATA port. I tried once with an adapter cable and fried the southbridge. You can plug a SATA drive into a SAS port though.
It could be argued that your RAID card should be the most expensive component in your server.
(Random anecdote) I once setup two 15K RPM SAS drives in RAID 0 for a mass ghosting operation and we easily saturated the gigabit switch. The photo of that very drive (3.5”) is still used in the Wikipedia SAS article.
Back up frequently and if a drive goes you can just reinstall on the 2nd drive. Doubling the number of drives to make it RAID 10 may be a bit pricey especially considering you won’t see a performance increase. Compared to RAID 0, RAID 5 will slow you down and cost more but you lose the single point of failure.
Motherboard/Case
I bought an ASUS based barebones kit. I’m happy that it can boot from the PCI-Express slot, not happy that it doesn’t support all of my RAM. The ASUS P3-P5G33 might be a better bet if you want support for more than 3Gigabytes of RAM. I’m not saying this is the best option but it works and it’s fairly inexpensive.
Un-interruptible Power Supply
Get a UPS, you’ll sleep better. Power surges aren’t the real issue. We have a big AC unit that kicks in and dims the lights. Mini brownouts ala Sim City. You don’t want that stress on your server. Also, get a Kill-A-Watt or something similar so you can see the power draw and don’t accidentally exceed the rated battery capacity of the UPS.
Cost – On a budget
Barebones PC – $200
4Gigs of RAM – $170
CPU – $230
UPS – $100
————
$700
Storage System:
SAS/SATA RAID card for the PCI-Express Slot – $300
Two Raptors or one Seagate Savvio – $350
————
$650
=====================
$1,350
Cost – Expensive, bang for buck
SAS/SATA RAID card for the PCI-Express Slot – $300
2 x Hyperdrive4s = $8,800 (32GB and 77,000? IOPS)
———–
$9,100
Some Predictions:
* RAM based SSD drives are going to get more popular as people realize that IOPS are more important than drive capacity or peak read performance in web servers.
*Someone will eventually release a RAM Based storage system with SATA-2 support that doesn’t require ECC memory. At that point, for under $2,000, you’ll be able to build a 32GB RAM drive with something like 80,000 IOPS and transfer rates of more than 220 MB/s, without the need for a RAID card.
* People will stop complaining about how slow Vista is. Maybe Microsoft should release this hypothetical device so bloatware truly no longer matters.
*Once that happens hard drives will find a niche as mass storage devices.
Flash based SSDs will remain hugely popular in mobile devices and laptops due to low power consumption.
* Power supplies will start coming with built in batteries which can power your volatile memory if the power shuts off.
* As performance becomes nearly free reliability will become a more important factor when buying a server.
* LAMP setups will become more proactive in tuning themselves. This will remain application dependent but a my.cnf generator based on your system specs will probably emerge. The bottleneck will become application design.
* Caching systems will vanish due to their complexity as drives cease to be the bottleneck (except maybe in very large systems).
* As performance ceases to be an issue we’ll see all sorts of interesting new applications. If Web 2.0 is/was about harnessing the knowledge of the masses and your database server chokes when it’s trying to write to the DB.
*Database performance consulting will become unnecessary because even sloppy code will run quickly. If your page loads in .004 seconds instead of .02 seconds nobody will care.
*I get the feeling Canonical(Ubuntu people) will eventually sell prebuilt LAMP servers. Fonality used to give away Trixbox as a Linux distribution but they recently started selling a pre-built phone “appliance” in addition to support.
* Future hard drives will be large PCI-Express cards populated with bunches of 4GB memory modules running at the speed of the PCI-Express bus. Current RAID cards have upgradeable memory modules for caching purposes. What if you had a virtual hard drive the same size as the RAM cache on the card? (ed. I was close on this one. Check out the video)
Some Random Thoughts:
*The Gigabyte RAM disk was pretty popular and insanely fast but no follow up was ever released in spite of consumer demand. Possibly because you could build an insanely fast storage system with a few of them in RAID which would completely devastate the enterprise hard drive market in a few short months. Yeah that sounds like a conspiracy theory but whenever I write about the HyperDrive4 I get a bunch of blog visitors from web mail accounts.
If Houndwire.com gets some traffic I’m going to push for some help for adding new features and a maybe a HyperDrive.
Networking:
Connectivity – T1s cost about $400 a month but you get 1.5 megabits upstream. Downstream isn’t great but you want to send out web pages not download mp3s right? Use someone else to host your images if you have a lot of them. No sense clogging up you T1 with big jpegs. You can host from a residential internet connection while you’re prototyping. Just be prepared to update DNS and configure port forwarding (and violate TOS). I used DD-WRT on a spare Buffalo to create a wireless bridge and hosted my little LAMP Laptop at home for a while unbeknownst to the guy who’s name is on the bill.
Use solid 10/100 switches instead of fast but potentially flaky gigabit switches, especially for DMZ switches where you’re not going to exceed a few Mb/s anyway. Don’t make your own cables unless you’re a masochist. You can get a multitude of colors cheap from NewEgg.com, or TigerDirect to stay organized. Have a couple crossover cables handy for computer to computer connections (IPCOP -> Server).
Firewall – I’m using IPCop because we had a spare PC lying around. IPCOP is like DD-WRT on crack without the wireless settings. Smoothwall is a cousin of IPCOP and a little easier to get going for the beginner but I prefer IPCop(you don’t have to register to read the manual). If you’ve had a big fancy corporate firewall before IPCOP will disappoint you. My problem was putting the web server on the DMZ with a real IP address. IPCop’s ORANGE zone can NOT handle using an IP on the same subnet as the RED. There are ways to hack it by putting the DMZ IPs in the RED zone and port forwarding to ORANGE but even then you’re only addressing the one IP problem. I’m going to look into DevilLinux which is apparently better suited to non residential configurations with multiple real IPs. The alternative is paying $700 for a 3 port SonicWall. DIY may not be worth your time, especially if you don’t have a spare PC to cannibalize.
Ideally you’d find an old laptop and put a cheap solid state drive in it for reliability. Old ToughBooks are great if you can find one. I once bought a flash to IDE converter and installed IPCOP on a camera sized memory chip. No moving parts means it’s cooler and less likely to fail. It’s still running as far as I know.
Software
For the sake of simplicity get Ubuntu Server Edition. It has a LAMP option when you’re installing which saves a lot of time. Get the 64bit version unless you’re not sure if your CPU is too old.
Don’t be afraid to break stuff in the short run. If you have to reinstall your LAMP stack a few times it’s just practice for down the line when your overheating Raptor RAID 0 array bites you.
MySQL Performance Tuning
Add the code for benchmarking.
Stuff I’m running:
Apache 2
PHP 5
MySQL 5
64 bit v. 2.6 Linux kernel
SSH
Samba
Webmin
Webalyzer
PHPMyAdmin
Here’s a video of an overcaffeinated guy with an equally hype dog explaining the basics.
The linux top command is your friend. You can watch MySQL CPU gobble up less and less CPU time as you optimize things.
If Ubuntu Server edition comes with all of that stuff installed in addition to a functional LAMP stack they’ll have a winner. Sell support to fledgeling startups using RedHat’s model.
Config Files and performance tuning:
Databases:
You can spend days tweaking my.cnf but the big gains early on come from making sure you’re indexing tables properly. If you’re running a query like “SELECT names FROM people WHERE age > 40″ make sure you have an index on the age column.
Radiohead’s new free album is killing the environment.
The Federal Reserve hasn’t hidden their absolute terror of a deflationary spiral. Bernanke is a student of the Great Depression and has vowed to never let it happen again. Greenspan and now Bernanke slash rates at the first sign of trouble because they can simply gloss over inflation with a questionable CPI. People are less forgiving when they suddenly lose their job when compared to paying more for eggs. And the foundation of this whole crazy debt laden financial system relies on constant inflation.
If the new Radiohead album does well the music industry as it currently exists is pretty much toast. Music is the first to go only because songs can more easily be squashed into small files. Movies will go the same route eventually. The deflationary forces of technology are no doubt freaking out the Federal Reserve. In fact, Ray Kurzweil’s new AI book has an entire chapter on deflation. At least in the ’30s entire industries weren’t forever vanishing, people just bought less for a while. If music goes free there will be no CD revival in five years.
So here’s my justification for the thesis. The Fed is watching the deflationary forces of technology bring down massive industries. They’re also watching the end of the housing bubble. The trillions of dollars recently created flowed to housing prices instead of food, energy, etc. Housing prices aren’t used when calculating inflation. The end of the housing bubble means a couple of things. One is the need for a new asset bubble which can hide inflation, the other is the risk of deflation due to the tsunami of foreclosures we’re witnessing.
The powers that be are currently worried or they wouldn’t have cut rates 50 basis points. Environmental regulations are only going to put downward pressure on the economy regardless of their potential long term benefits. If the Fed cared about the long term they wouldn’t have nearly completely devalued the dollar. As technology kills off gigantic industries you can bet the people at the Fed are telling the powers that be not to do anything that will destabilize this already battered ship.
Crappy holiday retail sales will be blamed on toxic toy imports so the cheerleaders at CNBC don’t have to admit housing is affecting the greater economy.
The housing market is about to get a lot worse. There are still believers but they’re quickly running out of money. Mortgage resets get really ugly next year.
Ipods/MP3 players of the future will run bit torrent and cache streamed audio over the internet. Radio stations will consist of math majors working on recommendation algorithms based on geography and local information like concert info will have to be peer to peer because custom playlists mean songs can not start at the same time. And it will be prohibitively expensive to distribute that many custom feeds from a centralized location. DJs won’t talk over songs (it’s more expensive anyway because they have to do it live) and every 20 songs or so you’ll hear the latest local music news.
I don’t agree with everything Ron Paul says but I agree with enough of what he says. You can bet on who will win the Republican nomination. People with money on the line at InTrade are giving him a 6.1% chance of winning. That’s incredibly high considering his small government views. My assumption is that most Americans want bigger government for one of two reasons. To blow up foreigners they don’t like or to create more bureaucracy in an attempt to fix human nature.
This is going to be another quick and dirty guide for setting up static NAT aka 1:1 NAT aka 1 to 1 NAT on IPCOP.
As Paul Graham 
IOPS are a good measure of disk performance on databases. The average consumer grade drive can handle 100-150 IOPS. One 15,000 RPM Seagate Savvio is in the 300s. My Raptor RAID array can probably handle 400+. Now consider that a good CPU can handle nearly 100,000 IOPS. Super expensive RAM based drives are useful because they eliminate the drive bottleneck.
Back up frequently and if a drive goes you can just reinstall on the 2nd drive. Doubling the number of drives to make it RAID 10 may be a bit pricey especially considering you won’t see a performance increase. Compared to RAID 0, RAID 5 will slow you down and cost more but you lose the single point of failure.
