Tuesday, 12 November 2013

How do you increase FreeBSD processor performance by 10%?

Turn on powersaving! This may seem counter-intuitive. Old school admins typically turn off all power management in BIOS, and ensure their servers don't go into power saving modes for performance reasons. Today, this isn't necessarily true.


Intel Turbo Boost. This tech ramps up the single core clock speed in a multicore processor when it's able to (eg when other cores are idle) providing better performance. For this to work, it must be enabled in BIOS as well as OS. In Freebsd, that's handled by powerd (man powerd).

The best  practice now is to enable hiadaptive mode on powerd, which enables TurboBoost, with a secondary benefit of power saving. Some DCs are already billing customers based on power consumption (AIMS!), so this is good to have. That's like having your cake and eating it too.

 hiadaptive  Like adaptive mode, but tuned for systems where performance
   and interactivity are more important than power consumption.
   It increases frequency faster, reduces frequency less aggres-
   sively, and will maintain full frequency for longer.  May be
   abbreviated as hadp.

Edit your /etc/rc.conf,

powerd_flags="-a hiadaptive"


/etc/rc.d/powerd start

You're all set.

Switch-independent NIC teaming on FreeBSD

On FreeBSD, teaming/link aggregation is handled by the lagg kernel module (man lagg). What the man pages don't tell you (nor any other site I've googled) is, what exactly is switch-dependent and which isn't. This blog post does.

Failover - switch independent
FEC - switch dependent
LACP - switch dependent (you need switch trunking enabled if you're doing LACP with 2 different physical switches, but this does have the benefit of increasing server bandwidth available for multiple clients)
Loadbalance - switch dependent (same as FEC)
Roundrobin - you will not likely ever use this.

Ok, so I had a case today where I needed to connect a freebsd box to 2 switches for switch high availability support, so there was only 1 option - failover.


1) ifconfig em0 up
2) ifconfig em1 up
3) ifconfig lagg0 create
4) ifconfig lagg0 up laggproto failover laggport em0 laggport em1 <ipaddress> netmask
5) route add default <gateway>
6) $$$

In other words, both link 'share' a single IP. To be clear, It is not actually sharing per se, since only the first interface becomes the master port (em0 in the above example), and em1 will not be used unless and until em0 goes physically down. You can do an ifconfig lagg0 to see which port is active.

     failover     Sends traffic only through the active port.  If the master
                  port becomes unavailable, the next active port is used.  The
                  first interface added is the master port; any interfaces
                  added after that are used as failover devices.

                  By default, received traffic is only accepted when they are
                  received through the active port.  This constraint can be
                  relaxed by setting the net.link.lagg.failover_rx_all
                  sysctl(8) variable to a nonzero value, which is useful for
                  certain bridged network setups.

     fec          Supports Cisco EtherChannel.  This is an alias for
                  loadbalance mode.

     lacp         Supports the IEEE 802.3ad Link Aggregation Control Protocol
                  (LACP) and the Marker Protocol.  LACP will negotiate a set
                  of aggregable links with the peer in to one or more Link
                  Aggregated Groups.  Each LAG is composed of ports of the
                  same speed, set to full-duplex operation.  The traffic will
                  be balanced across the ports in the LAG with the greatest
                  total speed, in most cases there will only be one LAG which
                  contains all ports.  In the event of changes in physical
                  connectivity, Link Aggregation will quickly converge to a
                  new configuration.

     loadbalance  Balances outgoing traffic across the active ports based on
                  hashed protocol header information and accepts incoming
                  traffic from any active port.  This is a static setup and
                  does not negotiate aggregation with the peer or exchange
                  frames to monitor the link.  The hash includes the Ethernet
                  source and destination address, and, if available, the VLAN
                  tag, and the IP source and destination address.

     roundrobin   Distributes outgoing traffic using a round-robin scheduler
                  through all active ports and accepts incoming traffic from
                  any active port.

Tuesday, 16 July 2013

HOWTO: Web Application Scanning

So you think your web app is vulnerable. How do you find out more using automated tools? Let's start with the freely available ones.

1) Nikto.

On FreeBSD,

cd /usr/ports/security/nikto && make install clean && rehash
nikto -update
nikto -host http://yourserver

You're looking for 0 errors.

2) Netsparker Community Edition for Windows

Download it here. Unlike some other free scanners, this can detect SQL Injections as well as XSS attacks. Highly useful.

3) N-Stalker Free Edition.

Download it here. Has limitation of up to 500 pages max, and reduced number of rules compared to paid ed.

LibreOffice vs OpenOffice, which to use?

One word...LibreOffice. Why?

  1. LibreOffice has 350 programmers and 20,700 code commits over the last 12 months, versus 50 devs and 4,900 commits for OpenOffice. While more programmers is not necessarily better coding, an active base of committers is a sign of a healthy project.
  2. Licensing. LibreOffice is based on LGPL, OpenOffice on ASF. ASF does not accept contributions from non-ASF licensees.
  3. AMD have thrown support behind LibreOffice, promising hardware acceleration to boost speeds. If you're an Excel power user, this will make LibreOffice a viable consideration as far as speed goes.
  4. LibreOffice will have mobile support. Android & iOS ports are being worked on, though it's still early days yet.

Download LibreOffice now.


1) http://www.theregister.co.uk/2013/07/16/libre_office_hardware_acceleration/
2) https://wiki.documentfoundation.org/ReleasePlan#4.2_release
3) http://www.ohloh.net/p/libreoffice
4) http://www.ohloh.net/p/openoffice

Saturday, 8 June 2013

FAQ: Which is faster and more reliable, wired or wireless?

As of today, June 8 2013, my stance has not changed. If you value reliability and speed in an office environment of any significant size, USE WIRED connections!  Wireless is no substitute. Useful Complement yes*, Substitute no. This is a no-brainer. Any " IT guy" professing otherwise should be immediately fired and clobbered with an Airport Express. This article is in fact targeted at you, gentle business owner, so you can make an informed decision.


WAP -  Wireless Access Point. That bit of kit that has 2 or more antennas sticking out the rear.
802.11n  - Wireless Networking standard. Transfers up to 150Mbits/second i.e. 15Mbytes/second. If you're very lucky.
10/100 Ethernet - Wired network standard where one can transfer up to 100Mbits/s, or 12.5MBytes/second.
10/100/1000 Ethernet - Wired network standard where one can transfer up to 1000Mbits, or 125MBytes/second. 

Consider the following.

You move into a spanking new office lot. Your IT vendor goes in, does a site survey (that's a pretty rare occurence right there, but let's use our imagination), and says, "let's go 11n wireless!" Upon which you send a PO for one or more Wireless Access Points to said vendor. Come installation day vendor does speed test (that's another rare occurence), and you exclaim "300Mbps? I'm not even getting half of that!" You, dear business owner, have fallen victim to a vile lie...that of believing advertised specs.

Fact 1:  There is no such thing a 300Mbps throughput by 300Mbps advertised equipment. It's a lie. It's a bigger lie than advertising a 2TB HDD that has only 1.8TB available (let's not go there just yet).

Fact 2: 300Mbps is *thereotical* output, not practical. Even assuming best case scenarios (assumption: no other WAP nearby to interfere, no other sources of interference, direct line of sight between client device and WAP,  both adjacent to each other, no other devices sharing wireless), you're not going to get even half that.  If you get 50% of theoretical, that'd be outstanding.

Fact 3: Wireless is a shared medium. 1 client device may get 150Mbps of throughput, 2 devices would get 75Mbps. 10 devices would get 15Mbps of bandwidth. That's almost as slow as networking technology in the early 90s. The only way to alleviate it is to have more WAPs eg 5 PCs on WAP1, the 2nd bunch of 5 PCs on WAP2, thus load balancing. However there is a problem. Wireless channels are very limited. Which brings me to:

Fact 4: Wireless channels are very limited. For the most popular 2.4ghz wireless band, there are only 3 distinct, non-overlapping channels..i.e. channel 1, 6 and 11 (to keeps things simple, Japan can use 1,5,9,14). 3 wireless channels for everyone. Once you start overlapping channels (i.e. putting channel 1 and 2 next to each other), you get what is known as Adjacent Channel Interference and performance is shot to hell. In fact, it's better to have to 2 APs sharing the same channel (eg both Channel 1) causing Co-Channel Interference, than to have  different but overlapping channels. Co-Channel interference is anticipated and managed by design, and performance is maximized. Adjacent CI isn't. Details, read here (PDF) . Also, most WAPs have a setting called "Auto Channel Selection" or "Clear Channel Selection". This is utter rubbish and usually causes Adjacent Channel Interference! Do not use this! Take out your android smartphone, fire up Wifi Analyzer, and check which is the best signal for your location. Manually enter this channel number into your WAP. And check again in 6 mths time. If you have a choice, use the less popular and more expensive (for now) 5Ghz band. There are 23 non-overlapping channels to choose from. 5Ghz though has one major downside: it doesn't penetrate walls very well, so if you have a multistorey building, you may need multiple 5Ghz WAPs for full coverage. Still, you'll thank me for it.

Fact 5: WAPs are hopeless beyond 30 clients. Even with 30 clients, network bandwidth will be an issue for anything but the most basic tasks (internet/email).

Fact 6: A site survey may give a clean radio environment, NOW. At this very moment. However  tomorrow you  may have a new office opening up next to you, and they just happen to bring along 3 WAPs from their old office. If you're in any kind of multistorey commercial building, you could very well have 40 APs in radio range, all fighting for the same 3 channels in the 2.4Ghz range, or 23 channels in the 5ghz range.

Fact 7: Wireless performance depends on how polluted the radio environment is. And there is a LOT of competition for the 2.4ghz range. Cordless phones. Baby monitors. Motors. Microwave ovens (since that's the frequency best to excite water molecules). Bluetooth devices. All of these will conspire to make your wireless network slower than streamyx.

Fact 8: If you're doing file transfer regularly across the network, or opening files from a server, or running an accounting or any application off a network drive, wireless is not for you unless you're a zen master with unlimited patience. Just don't do it. The difference in speed can be ridiculously dramatic. What may take 1 minute on wireless could take 5 seconds on wired.

Fact 9: We've been talking about performance and reliablity thus far. But wireless is inherently LESS SECURE than wired. For someone to hack your wired network, they'd need to enter your office and physically plug in their sniffer. To hack your wireless network, they just need to exploit your misconfiguration from the adjacent office, or even up to a few hundred meters away from the main road. And believe me, hacking wireless networks is pretty easy. Let me count the ways:

  • a) Open networks - running a WAP without having a password. Pretty rare nowadays, no one's that ignorant. Once upon a time, well intentioned folks left the APs open, so their neighbours could use it when they needed to. No good deed goes unpunished nowadays.
  • b) Using WEP encryption - this is totally hacked, there are tools that can hack WEP password in less than a minute.
  • c) Using WPA/TKIP encryption - hacked. Bit more difficult than b), but still do-able. Rainbow tables are making this easy.
  • d) WPS - supposed to make joining to a wireless network easy, but made it easy for hackers too.
  • e) AP SSID spoofing - hackers pretend that their rogue AP is yours. 
  • f) etc etc etc. 

*For most normal networks, where you don't need to worry about alphabet soup compliance such as PCI-DSS and HIPPA, you can just use WPA2-PSK or WPA2-Enterprise (for now anyway). Easy to use, moderately secure.

For secure networks, that's not enough. The best practice in that scenario is to put APs into their own untrusted subnet, and use VPN to access office LAN resources. You layer on your security that way. Even if the WPA2 scheme was proven insecure, you'd still have VPN to protect your traffic.

Technologies available for wireless

802.11n - 5ghz - USE THIS frequency if you have a choice. Unfortunately, plenty of laptops aren't compatible with it. The good news is that newer devices like the Ipad2, Samsung S3, Galaxy tab are 5ghz compatible. Buy a AP that supports dual radio, not dual band. Dual radio allows the use of 5ghz and 2.4ghz simultaneously. Dual band, you pick one band. Dual radio = GOOD, future proofing.
802.11n - 2.4ghz - the defacto vanilla standard today
802.11g - 2.4ghz - dump these.
802.11a - 5ghz - Old and slow, but mostly reliable. Never caught on. 
802.11b - Get rid of this device immediately, this screws up 11g and 11n transmission speeds.

Around the corner, 802.11ac draft standard (here we go again). No devices in market yet.

My background: Dealing with dozens of wireless setups by incompetent vendors who leave things in default. Starting messing with APs at Intel in 2000/2001 when Intel came out with their line of enterprise APs. Currently have a rooted android with firesheep and Wifi Kill. Purely as a teaching tool of course.

Monday, 3 June 2013

SPAM: The fight between Good and Evil

Spam. Junk mail. Whatever you choose to call it, the problem is an ongoing war between spammers and antispammers.

What the spammer has in their arsenal:

1) People whose PCs are compromised, thus getting drafted involuntarily into spammer botnets, which are large groups of PCs controlled by the botmaster. For spammers without botnets, they can typically hire a botnet from a botmaster for their spam run.
2) User ignorance. These are folks, well intentioned or otherwise, who unwittingly get their PCs infected, and having it become part of a botnet. If they're lucky that is. Some machines become child porn hosting sites.Were law enforcement to trace the IP address, the victim's IP address would appear as the culprit. Penalties can include jail time.
3) Unlimited supply of blissfully ignorant users, who see popups etc appear on their PCs, and odd behaviour manifesting itself  (eg clicking on google doesn't go to google). As long as they can still use the PC, they'll ignore it as long as possible.
4) Thousands of compromised PCs with free bandwidth. Bandwidth that can be used to spam or to take down the sites of people they don't like with DDoS (distributed denial of service) attacks .
5) People who read ads like "Make $Money$ fast at home!" and think nothing's wrong with spamming as long as they make a buck.

What the antispammer has in their arsenal:

1) Antispam software, the most famous of which is SpamAssassin. Bayesian Statistical Analysis.
2) What was once the fought by little outfits like Spamhaus, we now have folks like Microsoft (!!!) who are now going after spammers across the world. We can forgive Microsoft for Vista/Win8/WinPhone8. Hmm, maybe not Winphone8. Then again, one can argue Microsoft themselves make spamming easy by making Windows so easily compromisable, even a fully patched Windows8.
3) Moral rectitude. Being on the side of everything that is good and holy vs the greedy money grubbing immoral spammers. Sometimes it's just that clear. 

What exactly is spam?

One lax definition is any unwanted mail. Another is Unsolicited Commercial Email (UCE). Still, if you're in business, you do expect commercial emails from new and potential clients, even if it were unsolicited. A better take would be as per Spamhaus' definition, that it has to be Unsolicited AND Bulk email.

Unsolicited Email is normal email
  • (examples: first contact enquiries, job enquiries, sales enquiries)

  • Bulk Email is normal email
    (examples: subscriber newsletters, customer communications, discussion lists)
How do we nail these clowns?

Most folks don't have resources to nail these miscreants. The nailing can only be done with law enforcement across country borders, since zombie botnets frequently span countries making law enforcement difficult if not  nigh impossible. Only folks with the pockets of Microsoft can do these. That's not to say we cannot do anything...we can. The most important of which is to maintain clean computing practices. And I don't mean wiping your LCD screen every day.

Surely there must be some laws? Can't we just nab the filthy scum?

Assuming the spammer is local to Malaysia, we have some laws. In the Malaysian context, spam is categorized under Section 233 of the Communications and Multimedia Act 1988. The penalties are a fine <= Rm50k, imprisonment <= 1yr  or both. However seeing how the Malaysian legal system seems more interested in arresting people for posting uncomfortable Facebook comments or prosecuting online websites for posting news unflattering to the govt of the day, rather than doing something actually useful, I don't see this happening anytime soon.

The new Personal Data Protection Act of 2013 also has some interesting legal tools against the would-be spammer. Anyone disclosing personal data to 3rd parties without notification/permission would be liable for a fine of up to RM300k and up to 2 years' jail. So you cannot create an email database, then sell it to someone else for $. At least, that's my impression as a lay person.

Thus far, I'm not aware of a single case prosecuted by the Malaysian Govt under the CMA as relates to spam and I don't expect them to do so anytime soon. Still, the best laws prevent at best 5% of spam, whereas the worst antispam technologies prevent 80%. Let's hear it for technology.

As a user how do I minimize my spam?

Practise safe computing!

1) Don't get your PC/mobile infected!! Once infected, your device is compromised. Any and all information on it has potentially been leaked to the botnet owners. Change passwords immediately! DO NOT KEEP USING THE PC. GET IT REFORMATTED, NOT CLEANED, since most tech folks don't do a proper job cleaning. Frequently cleaning takes longer than a clean install, and your PC will perform faster with a clean install anyway. The keywords to remember here are: CLEAN INSTALL, from known good media (not pirated CDs!).
 2) Stop installing free apps that read all your contacts (and email addresses), unless they're from trusted folks. How do we know who is trusted and who isn't? Therein lies the dilemma. It's probably safer to trust an app with 1M downloads, vs an app with 20 downloads.
3) Do not use an administrative account (even with UAC) for normal work. Create a limited user account. This will minimize the damage any malware can cause, since they can only run at same privileges as limited user.
4) Do not use freemail (I'm looking at you, YAHOO and OUTLOOK.COM (what was formerly hotmail). Have your own domain (somecompany.com) with your own hosted email.
5) Do not disclose your email to every Ah Chong, Ahmad and Ramasamy! Keep multiple email addresses. Your  primary business card should have one email address for work only. DO NOT USE THIS ON MAILING LISTS!!!  OR GIVE IT AWAY FOR A FREE DRAW. OR AT ANY EVENTS/EXHIBITIONS. Keep a 2nd card for this. Your work  email is for clients (current/potential) only, those that you cannot afford to block.
6) Use content filtering to block all banner ads. Banner ad providers are frequently targeted by malware authors since these ads have a large audience. If the ads were hijacked, they'd have a large pool of PCs to infect and command. 
7) Use content filtering to block all known malware sites
8) Use an antivirus on every consumer device. Whether it's a Mac, PC, smartphone or tablet. Or a PC-based DVR. They're all potentially vectors for infection.
9) Porn sites are the largest infection vector, next to religion websites apparently (!). The former probably because the big brain isn't thinking clearly, and the latter perhaps because the admins have not the expertise to harden the sites (if you're in the latter category and based in MY, contact me for free advice and pentesting).
10) Don't ban the internet! Some folks are under the mistaken assumption that if their staff does not have internet, the PC won't be infected. Please note that USB flash drives are easy avenues of malware transmission, unless you superglue all USB and network ports. The better plan is to limit internet site access (eg blocking chat, forums, social networking), while allowing update sites for antivirus and Windows Updates etc so the security patches are installed in a timely fashion.
11) Use microSD cards with USB adapters instead of normal USB flash, since microSD has a write-protect tab (a long while back regular USB flash had too). If you copy a file to a suspect PC, ensure the write-protect is enabled so your card does not get infected.
12) Update everything! Windows Update (every 2nd Wed of the month), antivirus (check it has daily updates), Adobe Flash and Adobe Acrobat (the two most insecure software ever), Oracle Java. Adobe deserves a special place in hell, for wrapping their Flash Updater with trial software, current McAfee Scan Plus. Steve Jobs had the right idea...Don't use Flash if you can help it. If your bandwidth is limited, either use a proxy cache which will help with all mass downloads, or setup a Microsoft Software Update Server which helps with the Microsoft software. A client/server antivirus also helps if so, since only 1 machine downloads the updates and distributes them.
13) If you still use Win98, Win NT, Win2000 or an unpatched WinXP...well, you're on the side of spammers. Those PCs are wholly insecure and unsuitable for any sort of network environment.

If you use safe computing practices. you'll see perhaps 1-2% spam per day out of your total volume assuming minimal false positives (i.e. mail that is actually ok, but misclassified as spam). If you don't, well, you probably already have loads of toolbars and need to have your PC reformatted yesterday.

 Additional comments for wannabe Mail Admins

1) Greylisting - Nice idea. Doesn't work, when so many idiots run mail servers that don't follow RFCs. You end up troubleshooting other folks' mail servers. Not worth the headaches. Tried and discarded.
2) RBL - If you're trusting a 3rd party DNSBL to solely decide what mails YOUR mail server accepts, you're an idiot. These should only be used for scoring the mail, not an all-in-one accept/reject filter. The only exception IMHO is if these were properly managed IP reputation folks such as Postini/Brightmail etc.
3) Vacation mail - evil most of the time, even if properly implemented. For me, it's evil because it confirms the email address with information leakage (eg "I'm away  on vacation from the 23-27th of May, please stop by and rob my home since no one will be in").
4) Spamassassin (SA) has some default rulesets, however they need tweaking.On low end servers, sa-compile is excellent for reducing processor load.
5) Clamav with SaneSecurity - highly recommended. This reduces CPU load further since this does preliminary filtering on malware & phishing before the CPU-intensive SA kicks in for the antispam portion. Pure ClamAV doesn't work as well.

Wednesday, 15 May 2013

Why a UPS is mandatory for important equipment

Apcupsd logs taken from a client site in an industrial estate with frequent brownouts/power interruptions.

2013-01-29 16:58:28 +0800  Power is back. UPS running on mains.
2013-01-29 16:58:39 +0800  Power failure.
<snip dozens of lines >
2013-01-29 17:29:00 +0800  Power failure.
2013-01-29 17:29:02 +0800  Power is back. UPS running on mains.
2013-03-01 16:00:00 +0800  Power failure.
2013-03-01 16:00:01 +0800  Power is back. UPS running on mains.
2013-03-02 06:04:39 +0800  Power failure.
2013-03-02 06:04:41 +0800  Power is back. UPS running on mains.
2013-03-07 11:54:37 +0800  Power failure.
2013-03-07 11:54:41 +0800  Power is back. UPS running on mains.
2013-04-01 08:51:26 +0800  Power failure.
2013-04-01 08:51:28 +0800  Power is back. UPS running on mains.
2013-05-14 15:56:48 +0800  Power failure.
2013-05-14 15:56:54 +0800  Running on UPS batteries.
2013-05-14 15:58:08 +0800  Mains returned. No longer on UPS batteries.
2013-05-14 15:58:08 +0800  Power is back. UPS running on mains.

As can be seen, if there was no UPS, the server could have gone down multiple times per month. If the server was set to power on automatically, the server PSU could have been stressed to blowing point on the 29th of Jan, since power was going off/below normal threshold frequently.