Category Archives: Open Source

PHP MySQL Benchmark Tool (PMBT v. 0.2)

11 Replies

The old saying “Need is the father of innovation” (or something like that) held true this weekend. I was looking for an easy way to benchmark MySQL for some RAM drive InnoDB experimentation but couldn’t find anything cross platform, user friendly, and created after 2005. So I built an early version of what I was looking for.

This is a very synthetic benchmark for now. In some instances InnoDB is much faster than MyISAM (simultaneous reads/writes) but that doesn’t come across in these results. I’m planning on beefing up the benchmark options in later versions. This tool is currently useful in benchmarking hard drive / RAID performance when using InnoDB. It’s also good for basic my.cnf tweaking.

So here is the PHP MySQL Benchmark Tool (PMBT) version 0.2 (download). You’ll need InnoDB (optional) and MySQLi (might change this) support for now. Tested with MySQL 5.0.51b-community.

Current Features:

  • Easy to use: No compiling, complicated config files, or OS requirements. Web based.
  • Specify your table type (MyISAM, InnoDB, MEMORY)
  • Insert, Select, and Update testing (CRUd)
  • Specify number of records to test
  • Results displayed in operations per second (instead of total seconds)

Future Features:

  • Custom benchmarks on existing tables
  • Aesthetic Rejuvenation
  • Support for non mysqli connections (do a find/replace on the code for now)
  • Full text benchmarking for MyISAM
  • Less Syntheticish Testing (simultaneous reads/writes (AJAX?))

Instructions:

  • Unzip to a web accessible folder.
  • Change the DB settings in dbconnect.inc.php to match your setup.

Let me know in the comments if you find this useful.

And in case you’re curious InnoDB inserts went from 100 per second to 9,955 per second on the RAM drive. MyISAM tables saw virtually no change in performance. The InnoDB log was also on the RAM drive for that benchmark. Full ACID compliance at half the speed of a Memory/Heap table ain’t bad. I’m writing up a post on how I made it happen.

In other news, I added the LastFM widget over to the right, it plays songs based on my preferences from my music library.

WordPress Blog Migration Thoughts

Leave a reply

If my blog seems slower lately there is a good reason. I migrated from a dedicated quad-core server to a free WordPress.com account and finally to BlueHost.com ($7/month). The following is a quick look at the pros and cons for people in the same boat.

WordPress.com
Pros:

  • Free (myblog.wordpress.com).
  • Allows use of a custom domain (optional – $10 a year).
  • Allows post import from existing blogs using the import/export tool (see caveats below).

Cons:

  • Very limited use of custom templates and plugins. If you have a real WordPress install you’ll need to make some sacrifices.
  • Import tool is slow and confirmation is delayed.
  • Slow at times, no tech support.

Hosted Shared Servers (BlueHost, DreamHost, etc.) list of more
Pros:

  • Allows a full installation of WordPress and other blogs / content management systems.
  • Much better control of the server, typically with something like CPanel or Webmin.
  • Ability to create multiple databases.
  • Use of custom domain (myblog.com).
  • Mail account support (me@myblog.com)

Cons:

  • Cost $7-$10 a month. That’s well worth it in my opinion given the limitations of free solutions.
  • With BlueHost I experience occasional slow downs because I’m using a shared server.
  • Might be confusing if you’re not comfortable working with databases and WordPress config files

Migration Tips:
Do NOT rely on the export tool that comes with the default WordPress install. I backed up using the export tool as well as the WordPress Backup Plugin. It’s a good thing too because the export didn’t work right and I would have lost most of my blog if I didn’t have a backup of the backup.

BlueHost limits your ability to name databases. I ran into the problem where my wp_config file needed to be changed a little bit so WordPress could see my restored DB considering its new, BlueHost compatible, name.

I used the WP Backup Plugin tool but the import didn’t work. It dumps a big SQL file which you can import directly into a new DB using the tools provided in CPanel. You’ll then need to install WordPress after you’ve created a new DB and tweak the config to connect to the DB. At that point you’ll be up and running but you’ll need your WordPress API Key to get some of the plugins up and running again.

Hard Drive Progress and the Viability of Vista

2 Replies

The slowest, most unreliable, and out-dated hunk of machinery in any computer, Mac, PC, laptop, whatever, is the hard drive. Computers today are like incredibly hi-tech cars driving around on wooden buggy wheels. Millions of dollars are poured into developing faster engines and fancy suspension systems but much of it goes to waste because the wooden wheels can only handle speeds under 30 miles per hour.

Hard drives are basically just shrunken down record players. They are no doubt marvels of engineering but they suffer from an unavoidable need to physically move an arm around, reading and writing data onto a spinning hunk of metal (see video). That’s all changing thanks to the emergence of solid state drives. These are a lot like the USB thumb drives stuck to countless key chains except they’re bigger, faster, and you can install your operating system on them.

Next Level Hardware just reviewed the latest generation solid state drive from MTron and it gives a really interesting glimpse into software performance on computers that are no longer bottlenecked by wooden wheels. I won’t get into the geeky details but the thing basically boots Vista more than twice as fast as the fastest mechanical hard drive. I had Vista on a pretty capable laptop with 2Gigs of RAM and it would take minutes of watching the hard drive light blink before I could actually use the thing. But what if Vista wasn’t such a slug? Would it be worth another look?

These drives are horribly expensive but prices are dropping fast. The NLH review is interesting because it’s now possible to imagine a world where there are no more computer bottlenecks and reliability is no longer a concern because of the durability of the new crop of drives.

If performance and reliability are no longer issues some obvious questions come to mind: Why would anybody ever buy a new computer? Will Vista finally find acceptance once the performance issues are solved by better hardware? If nobody needs to upgrade because even cheap PCs are just good enough will the hardware industry collapse?

I predict that people will continue to buy new computers because the cost will be much lower as we move towards system on chip processors from Intel, etc. near the end of the decade. Got a virus? Just buy a new computer for $40. Storage will be remote if Google keeps its promises so even a computer failure will cease to be a big deal.

I’m still skeptical about Vista. By the time these solid state drives are cheap enough for mainstream use Vista will have been declared dead. There are rumors that MS is working on a really solid new OS that could replace Vista but Linux distributions may be good enough by then that it won’t make sense to pay an extra hundred dollars for a DRM shackled OS. Apple is interesting because standards are making the OS of choice nearly irrelevant but they’re also opening up Apple to competition from Linux. Virtual machines are probably the most interesting development because they will make applications OS agnostic. Want to run Office 07 in Linux or Mac? No problem, just boot XP in a virtual machine and do what you need to do.

Soon computers will be nearly free, performance will be more than anybody can use, at least for today’s apps, and the prospect of spending $150 for an operating system running on a $100 PC seems unlikely. Technically, economically, philosophically, Linux would seem destined for world domination.

OpenID + WordPress = Finally Working

8 Replies

OpenID is basically an attempt at single sign on. In theory you’ll be able to go to any site and log in or comment without the need to remember 31 passwords and user names.

So far it’s working great. I have it setup so that I just have to type in my blog’s url unbeknownst.net, it remembers my password, and I can log in and comment at a growing number of sites. My next goal was to add my blog to said growing cadre of OpenID compatible sites.

The best option currently looks like the WP-OpenID plugin. It was a pretty straight forward install but for some reason it was giving me blank pages when I submitted a comment at first. Seems to be humming along now. Feel free to try it out with your OpenID if you dare. I’ll leave the first comment using my OpenID account.

I allow anonymous comments on this blog so it’s not a big deal but you do get to bypass the spam filter if you’re using OpenID (at least for now).

* Update: Delegation doesn’t work. On other sites I can use my blog URL to log in but this plugin won’t let me do that for some reason. I’m sure they’re going to fix this in an update one of these days.

IPCOP with Multiple Static IPs (1 to 1 NAT)

2 Replies

This is going to be another quick and dirty guide for setting up static NAT aka 1:1 NAT aka 1 to 1 NAT on IPCOP.

Commercial grade routers are expensive beasts. A run of the mill Sonicwall Pro Firewall with DMZ support with set you back well over a thousand dollars. They’re nice but you also have to pay an extra monthly fee if you want fancy features enabled.

So you have an old PC lying around and a couple of spare network cards, or maybe you get Soekris 4801-60 ($260 see photo). And you’re thinking to yourself “Well, IPCOP is a free download, I just saved $1200 dollars!”

Not so fast buddy. IPCOP can do most of what my fancy Sonicwall did but you’re going to have to get your hands dirty.

Good Cop:

  • Free
  • Very powerful on a good computer
  • Intrusion detction using Snort (not intrusion prevention)
  • DNS caching, transparent proxy, lots of graphs, basic QOS
  • Basic VPN support
  • Open Source

Bad Cop:

  • Not very intuitive, can be frustrating to install
  • Advanced features require add ons and text file editing
  • Intrusion detction only detects, you need Guardian to beat up the intruders
  • DMZ setup is quirky, no transparent DMZ mode
  • Tech support limited to forums, I used to know the Sonicwall guys on a first name basis

These are the basic steps involved in getting a firewall running including a DMZ for your servers with multiple real IP addresses which will be separate from your green LAN zone. So this is a Red, Orange(DMZ), Green(LAN) setup.
* Set up aliases – This is pretty straight forward, read the manual. You’re basically telling the router about the static IPs associated with your internet connection here (at least those you’re planning on using). Our T1 came with 16 or so.
* Set up port forwarding – Again, just read the manual. Forward port 80 from your virtual IP addresses to your DMZ boxes.

Now you can visit your sites from the outside but if your server connects to an outside site it will appear to be coming from the RED interface on your firewall instead of the alias IP you set up. That’s a problem for a lot of reasons. Email, firewall rules from other machines, etc., start to freak out.

Here’s the fix:
* Turn on ssh (system, SSH status, enable)
* Use putty or whatever, to log into port 222 (not 22). (or just use the router’s keyboard/monitor)
* Edit your /etc/rc.d/rc.firewall.local file. It’s very important to put this in the right place. This should go beneath the line containing start) and above the line containing ;;

/sbin/iptables -t nat -I POSTROUTING -o eth1 -s 192.168.1.2 -j SNAT --to-source 23.23.23.23
Make sure you change eth1 to eth0 or whatever your red NIC is named. (go to status, network status and look for the red font). Also, change 23.23.23.23 to whatever the is external IP you want to use (same IP as used in the alias step above).

Reboot. Test it. On your DMZ server install lynx, the text based browser. For Ubuntu that’s sudo apt-get install lynx

At the command line type lynx -dump whatismyip.com
That should spit out something like Your IP is 23.23.23.23
If your DMZ server is running Windows just visit whatismyip.com in a web browser. It should return the IP address of the alias IP you configured earlier and NOT the IP address of your router’s red interface.

*note IPCOP will not let you simply set your DMZ servers up with real IP addresses and use transparent DMZ mode like you can with the Sonicwalls. You have to put them on a subnet separate from the red interface and port forward.

*note2 – Because IPCOP can’t support transparent DMZ mode you have to set the gateway for your DMZ boxes to the DMZ IP address of your IPCOP and not the one provided by your ISP.

*If you want to turn on intrusion prevention read the following words I found at Snort.org

Just to clarify, guardian is not an addon to ipcop, it’s just a program that read snort files and modifies the linux firewall using iptables. Once you get a good sample of your network traffic viewing snort logs, you should get a general idea of what to enable/disable in the SNORT rules. To test it, just run a port scan to the device, and then try to go into the internet from the same device. To make extra sure the blocking is done, you can vi the iptables file in /etc. You should see the ip that’s blocked. It’s not hard to set it up. What’s harder is to configure rules in snort and the ignore list of guardian.

Ubuntu Feisty Fawn Server & Desktop Thoughts

2 Replies

I had to replace my old LAMP web server due to a hard drive failure so I decided to give Ubuntu Feisty Fawn server edition a whirl. So far I’m pleasantly surprised. I had some requirements going in:

  • LAMP Stack – Needs to be an up to date, Apahce/PHP/MySQL based web server
  • Webmin – Because I feel emasculated when I plug a mouse into a server but the command line slows me down sometimes
  • PHPMyAdmin – It just makes life easier
  • SSH – Though the server is on a laptop with an LCD, etc., I prefer to config from other machinees
  • SAMBA – I develop with Notepad++, which I can’t get running in Linux/WINE, so I wanted to map a network drive, though BlueFish looks interesting
  • PEAR – Why reinvent the wheel if you’re developing PHP web apps?

Setup was a piece of cake for the most part. If I ever find time I’ll put together a howto combining all of the steps involved in building this thing. Webmin, PHPMyAdmin, SSH, SAMBA and PEAR should all be options in the installer tool at some point to make our lives easier.

As for the desktop version, it’s a huge improvement, at least on my Dell Latitude D600 Laptop. My resolution was correctly set without intervention (thank god – Xorg.conf is evil) and flash for Firefox was a piece of cake.

Wireless networking was the only thing that didn’t work. Because I’m a bit of a nerd I figured out how to get it working but it took some research. In case you’re curious, plug in a regular LAN cable to get online. Go to System > Administration > Synaptic Package Manager and click Search. Type in fwcutter and install the tool it finds. Then check Fetch and extract firmware and click Forward. Reboot and voilà, works for many of the Broadcom Truemobile cards.

This ease of installation is really remarkable considering the headache of an XP or Vista fresh install. I had one minor hurdle and now I’m one step closer to an entirely open source existence. Not that I’m opposed to proprietary but I prefer not to spend money if at all possible. Oh and whatever you do, do NOT turn on the desktop effects if you’re using a laptop with a ATI Mobility Radeon 9000.

Image is called Fawn, more info here

Inflation, Technology, and the Wealth Divide (and KDE and Open Source)

6 Replies

Not many economists can tell you what Moore’s Law is or what it could mean for the economy but I think the brains at the Fed probably know exactly what the following report from the Cato Institute implies:

“Machines complement human labor when they become more
productive at the jobs they perform, but machines also substitute for human labor by taking over human jobs. At first, expensive hardware and software does only the few jobs where computers have the strongest advantage over humans. Eventually, computers do most jobs. At first, complementary effects dominate, and human wages rise with computer productivity. But eventually substitution can dominate, making wages fall as fast as computer prices now do. An intelligence population explosion makes per-intelligence consumption fall this fast, while economic growth rates rise by an order of magnitude or more. These results are
robust to automating incrementally, and to distinguishing hardware, software, and human capital from other forms of capital.”

So my thinking goes like this:

  • Our economic system is built on inflation
  • Technology is advancing exponentially
  • Technology causes deflation (higher productivity)
  • The Fed is fighting deflation by increasing the money supply at the same rate technology is advancing
  • Inflation causes a misallocation of resources
  • Misallocation of resources is growing which results in asset bubbles
  • Inflation is a lot worse than it appears because it’s hidden by techno-deflation
  • Inflation can exist without prices rising much if the money supply is increasing
  • Misallocation of resources is a function of increases in the money supply, not increases in prices
  • CPI is fairly meaningless
  • If the Austrian School is right then this credit bubble is going to end painfully

The next two thoughts will probably end up as their own posts but for now have a gander.

Here’s an interesting article on the language processing features of KDE 4. If you’re not sure what KDE is think of it as the eventual replacement for your Windows desktop.

KDE 4′s Sonnet will turbocharge language processing
There is a good chance that in 10 years Linux will replace Windows as the de-facto-standard desktop operating system. There’s also a good chance that KDE will become the most popular window manager for Linux. Some cell phones are already running versions of linux so the following excerpt from the above article could affect a lot of people some day.

With the Sonnet library for KDE 4, developer Jacob Rideout hopes to reinvigorate the field of desktop linguistics by adding automatic language detection and other innovative features. Sonnet is to be for KDE 4 what KSpell 2 is for the current version of the K Desktop Environment, providing spellchecking facilities to applications as diverse as the Konqueror Web browser, Kopete instant messenger, and KWord office software. Unlike KSpell, however, it will also provide grammar checking, multilingual tools, and perhaps even translation, dictionary, and thesaurus functionality across all of KDE.

And if you thought Apple’s OSX was the zenith of desktop computing I’d recommend watching the following video. I demoed Beryl at work for some interns and they were stupefied, and probably a little more interested in computers.

Open Source as distributed intelligence contradicts the view that centralization is only possible with a few smart people in office. So when applied to government is centralization just the means to an end that we’ve gotten so used to that it became the end?

Interesting quote:
“Sometimes it is said that man cannot be trusted with the government of himself. Can he, then be trusted with the government of others? Or have we found angels in the form of kings to govern him? Let history answer this question.”
Thomas Jefferson

I think my next post will be on the last idea.

Idea as a read my laptop in my backyard. It’d be nice if someone invented a firefox plugin that would read highlight text with a right click. It’d need a playback speed adjustment tool and export to mp3 option. Then I could hilight an interesting story and dump it on an MP3 player and go for a jog. Or just sit here with headphones on.

Open Source and its Impact on Information in the Future

1 Reply

I posted this on Buzzmachine a while back and stumbled upon it again. I agree with myself here and am a little disappointed that I’d forgotten my point. This is one of those stream of thoughts I enjoy because it’s a little bit bigger than just technology. It is about information, the technology you just have to know to speak the language.

# KirkH Says:
October 26th, 2005 at 6:52 pm
What value does del.icio.us add? They allow me to manage my bookmarks from any computer in the world, you call it portability. Well what if someone created an open source WordPress plugin that replicated those features? In a few years, (sure this sounds crazy) when mobile broadband is ubiquitous, everybody will have a server somewhere (probably LAMP). Personal servers will be like cellphones are now though hopefully easier to use.

That means we won’t need Flickr or Del.icio.us to store our content anymore. Google will still have the ads next to its search results but it’s entirely possible that tagging (Flock could make it ubiquitous) will out do Google’s spam succeptable web crawler system. In fact Google hires people to spot check data. If you apply something like Reddit.com’s up/down voting system to del.icio.us then spam might just cease to be a problem (imagine a button on a web browser that you could click to flag a spam site). Or think Hot or Not but for websites.

So in 10 years when IPV6 is out and everybody can have a handful of IP addresses(phone numbers for servers) for next to nothing, and everybody has their own server instead of using a federation of web based services with quazi-open APIs, we will be in control of our own information. Netflix-esque recommendations will work for websites, photos, ideas, etc. and search engines as we know them would become useless when enough people are tagging. I already use del.icio.us more than Google to find stuff. Same thing with Flickr and images instead of images.google.com.

Web 3.0 (I know I know, and assuming the gray goo doesn’t get us) might be the same as web 2.0 except instead of using large corporate web apps, we’ll just apt-get the latest greatest version of OpenSource-Flickr and run it on our personal servers. This will probably start with plugins but it might eventually evolve into a big, well integrated application that handles blogging, tagging, photos, an open version of skype to store voice/video mail, bittorrent, me-mail server, and all sorts of preferences. It’d be far more economical to do this with shared hosting and stream content to whatever odd little devices we’re using at that time.

The advantage of centralization is that it’s simple to create something like the “Flickr Interesting Photos from the last 24 hours” page. The only way to solve that problem would be with some sort of newfangled peer to peer system assuming they’re not outlawed by then. So corporations will no longer have a monopoly on centralization.

And that is why I think Flock is a big deal. It brings tagging to the masses. If Firefox and/or IE encourage a user to create an account somewhere before bookmarks work the search problem takes care of it self.

My point is that we can regain control by using open source alternatives to the current big web2.0 apps on our own servers. An open source search engine is only possible if it takes advantage of the wisdom of the crowds. That’ll be possible when everybody starts tagging.

The Sunburned Onion

Leave a reply

Every once in a while I reassess my thinking in reagards to economics and peel off a layer of confusion. Post peel, my current thinking goes like this: The Austrians were right in that we’re currenly living in a massive credit bubble that’s bound to bust… eventually. When is anybody’s guess.

I still hold that the next credit correction will be swift and massive due to deflation as a result of accelerating technological progress. And I don’t mean chips. I mean emergent order in the form of global parallel decision making systems unleashed by the maturation of communications systems and software including open source stacks and TCP/IP. People who don’t like to hear themselves talk call it Web 2.0.

Anyway, I think this emerging meritocracy is going to crack the Fed’s serial bubble system. At some point the consumer is going to get exhausted as the smart money gets out at the top. I know a 30 year old guy here in San Diego. He flips houses and owns 7. That is not normal. The tech bubble probably victimized the CNBC watching investor class. The poor weren’t too affected. The investor class didn’t get Option ARMs. So I think we’re probably going to see a recession that just pummels the lower and middle classes. The Fed could cut rates but we might start seeing articles about liquidity traps. It’s not every day that you open Yahoo Finance and see an article titled “The Last Days of the Dollar”.

So in a sense I’m back where I started. If this bubble finally pops it could mean massive deflation. On the other hand, maybe the Fed can keep this beast duct taped together, leaching off the lower prices we should have seen already. In the land of blogs quotes temporarily gain fame. This one is all the rage right now:

“The market can stay irrational longer than you can stay solvent.”
John Maynard Keynes

But he also said

“Capitalism is the astounding belief that the most wickedest of men will do the most wickedest of things for the greatest good of everyone.”

No, capitalism is the belief that wickedest of men will get jobs as politicians.

So instead of Keynes I’m reading about Joeseph Schumpeter. The guy wasn’t dumb. From Wikipedia

From current thought it has been argued that Schumpeter’s ideas on business cycles and economic development could not be captured in the mathematics of his day – they need the language of non-linear dynamical systems to be partially formalized. Although he failed to become the most influential economist in history, he certainly ranks as one of the highest during the 20th century.

Other stuff:

  • I’m listening to Arcade Fire – Funeral – finally (released in 2004). Not bad at all.
  • I caught a fairly big name journalist pilfering stories from a blogger. No one else spotted it. As soon as I find time to move the blog to a new server I’ll publish it.
  • I’m working on an illustrated guide, how to install a T1 connection. Go here to see the photos.
  • I got busted for hacking the parking gate system at work with Asterisk. I would open the gate using our digital receptionist and a tone generator. The facilities guy was trying to be mad but he couldn’t help but laugh and ask how I did it.
  • I have a $2500 router at work. The manufacturer wants $600 a year to enable (read un-disable) a fourth port. So I’m just building a dedicated router with IPCop to connect the Cox LAN and T1 DMZ. Free software on an old Dell box with a spare NIC, can’t beat it.
  • My car’s brain has nearly undergone the transplant. I just need a couple more cables and some gumption.

    • HowTo – Free Phone Number and Long Distance

    2 Replies

    The Mission:
    My own personal free long distance phone system so I’m no longer beholden to “Universal Cost Recovery” and early termination fees. Oh and I want to replace the on hold elevator music with Tool.

    The Equipment:

    • A laptop running CentOS(Linux) and TrixBox / Asterisk (see photo)
    • A Polycom 601 SIP Compatible Phone (optional if you use a soft phone)
    • A Broadband internet connection
    • An account with Free Digits to get an incoming phone number.
    • An abundance of patience.

    My free phone number is (563)773-7469 (EDIT: Currently off line). If you call it the laptop in the photo will handle the call, ring my phones and take a message.

    I’ll be updating (and organizing) this post with the steps involved over the next week…

    *UPDATE*
    Made some changes and I can dial out now. Incoming calls aren’t working but it’s an Asterisk setting somewhere and not an outside problem.

    *UPDATE 2*
    Everything is now working except I’m unable to change the on hold music.
    Here are some of the steps involved in getting the box usable.

    Delete Zaptel Trunk
    Go to outbound routes and Trunk Sequence. Change it to SIP /freedigits
    Create a ring group (all phones)
    Setup the inbound route to use that ring group.
    Add Dial Patterns to outbound routes, long distance, etc.
    Allow Anonymous Inbound SIP Calls


    *UPDATE 3*
    On hold elevator music replaced with Tool. I’m maxing out the speaker on my Polycom 601 right now.