Vlado Chavkov's Blog

jQuery and YUI 3: A Tale of Two JavaScript Libraries

2010-10-28T20:20:00.001+03:00

jQuery and YUI 3: A Tale of Two JavaScript Libraries "

Recently I had the opportunity to build my first JavaScript front end application. What follows is a short story of the discovery and evolution that comes about when trying to use tools that aren’t suited for the job at hand. It is an account of the learning acquired through developing the same front end application using two very different libraries, jQuery and YUI 3. Details of the client and the project have been intentionally omitted.

Overview

The project involved the refactoring of several disparate Flash tools into a single interactive application based on open standards for a large content publisher. Of high importance, the application had to be highly optimised with a small initial foot print due the large number of daily page impressions the client receives. Several phases were involved, with the first being an initial proof of concept.

The concept involved the development of one view of what would be the completed application. It consisted of:

An initial seed file (< 1KB) responsible for the dynamic loading of any frameworks (e.g., jQuery or YUI 3) and the initial application file.

The development and inclusion of jQuery plugins to support form element styling and validation, and dynamic chart visualisations.

The generation and population of UI, based on user inputs, configuration defaults and the application’s location within the publisher’s site.

The calculation and presentation of information based on the user’s input.

In the interest of full disclosure, my own experience up to this point had been in developing small, standalone solutions, the type of which you could typically describe as plugins. These included dynamic UI components such as image carousels, interactive maps and Twitter / Flickr widgets. At the time of first dabbling with JavaScript, jQuery was popular, easy to learn and allowed me to quickly pick up the basics needed for the projects I was working on. However these were all standalone units and had no need to interact with other code or as part of a larger application.

First Attempt

On completing the first phase of the project, it became painfully obvious that I was dealing with a very different beast than what I was used to. Having had little experience in code organisation, my code quickly became disorganised, inefficient and repetitive. As a result, the first part of what would become a much larger application was slow to load. In fact it took eight seconds to generate that single proof of concept. A big change was needed and while I had considered using a small library like Dean Edward’s Base or John Resig’s Simple JavaScript Inheritance pattern to add class-based inheritance to jQuery, I decided to go one step further.

What I wanted was a complete, mature framework within which to develop my first OO application. Something that would effectively guide me through the process. In reviewing the available libraries I decided to adopt YUI 3 for the following reasons:

Integrated, inheritance-based application development support including attribute and class management.

Long term solution:
- Support for standards and accessibility.
- Funded by a large well known organisation, Yahoo!
- Associated with respected names like Douglas Crockford, Nicholas Zakas, and Stoyan Stefanov.

Performance optimisation:
- Initial seed file of only 7KB.
- Lazy-loaded modules on demand.
- CDN delivery.

Varied and comprehensive documentation:

Mature, consistent evolution between releases.

Integrated tools in YUI Compressor, YUIDoc, YUI Builder, and Console.

Not just JavaScript, a CSS framework too.

Take Two

Integrating YUI 3 brought several direct and indirect benefits to the project:

Inheritance-based architecture and class management through the Attribute interface, and Base and Widget classes producing performant, reusable and organised code.

Separation of presentation from model and data using the Widget class to render alternate views (inline or overlay) based on the application’s location within the site.

Sandboxing and dynamic module inclusion through YUI.use().

Cross-browser console debugging using YUI Console.

On save, performance optimisation using YUI Compressor in Eclipse.

Easy inclusion and integration of pre-existing jQuery plugins.

On save, automated code documentation using YUIDoc.

The final result was a happy client and a finished product with sub-second load times (remembering it took 8 seconds to load the initial proof of concept).

Lessons Learned

Select the right tool for the job

In reading this post I’m sure some readers will view this as anti-jQuery. Not at all. jQuery is a fantastic project responsible for many innovations. But, as with any tool, it has its strengths and an intended purpose. Its strength lies in normalising browser inconsistencies, lowering the barrier to entry for the novice and improving the efficiency of experienced programmers. The primary learning that came out of the project was that you can’t rely on one tool for every job. YUI builds on what jQuery can provide by also offering a well architected application framework. But it’s fair to say that it comes at a cost, see the next point.

Steep learning curve

You need patience, especially when writing your very first application with an unfamiliar library as I did. However the payoff is immense. By learning another library, not only will your core JavaScript skills improve, you’ll also develop a deeper understanding of how libraries work and the benefits they bring. I try to learn something new about YUI everyday and the more I learn, the more I’m in awe of the thought and sheer talent that’s gone into building YUI.

Only load content when you need it

While it’s certainly programmatically easier to just to load everything you may need upfront, the performance improvements gained as a direct result of lazy loading content only when you need it is huge. This was one of the key contributing factors for drastically improving the performance of the application.

Interact with the DOM as little as possible

This point doesn’t relate to the specific library used. By caching the required DOM elements and utilising HTML templates more, UI rendering time fell considerably. Nodes can be cached using Y.one() while node lists can be captured using Y.all(). Also Y.Node.create() was very useful in efficiently converting large text strings of HTML to DOM elements prior to insertion.

Learn by example, use a CDN

In using YUI’s CDN delivered library we decided to deliver all the project’s assets via CDN. This was probably the next largest contributing factor to the performance improvement.

Pub, sub hubbub

For those experienced programmers out there, try not to laugh at this one. Having been used to writing little more than plugins in the past, I had no idea how applications should communicate internally. Even after reading “Custom Events allow you to publish the interesting moments or events in your own code so that other components on the page can subscribe to those events and respond to them,” I still missed it.

As it turns out, YUI’s custom event publish-and-subscribe model works beautifully for inter-class and inter-object communication. You can even subscribe pre and post to events and include dynamic logic to suppress bubbling based on certain conditions.

Integrate best practice into your workflow

Using Eclipse we were able to integrate JSLint and YUI Compressor into the build process. Put simply, every time you hit Ctrl / Cmd + S your JavaScript code is validated and optimised. That means you’re testing against optimised, production grade code from the very first line of code. It also means that you won’t forget to optimise in the frantic final race to the delivery deadline.

Learning YUI on the Job

Although everyone has a different learning style, I thought I’d share what resources I used to learn YUI with a specific objective in mind.

Watch the relevant YUI Theater episodes to get a general overview of the library or learn a specific module. I can highly recommend starting with:

Read up on YUI on the Yahoo! Developer Network. I try to read a little bit every week and learn more each time I re-read it.

Read the API documentation. If you can’t find it on YUI Theater or on the Developer Network, dig into the API. It even pays to read the code directly.

Read and post questions to the forum on YUILibrary.com.

Play a lot and have fun!

Conclusion

YUI 3 is a full-featured, mature and constantly evolving library suitable for small to large projects. As front end web applications become more complex, the need for libraries like YUI will grow. While for the uninitiated it can be a daunting experience at first, if you stick with it you will be rewarded.

About the Author: Mark is a Senior Front End Developer at VI, a multi-disciplinary communications agency established in 1981 with a design orientation. Today the agency fuses its work in digital, direct and design disciplines to deliver measurable outcomes for a broad range of B2C and B2B clients.

tar — how to create and extract tar.gz and tar.bz2 archives

2010-07-25T12:29:00.001+03:00

Introduction

The commonly employed archive types on Linux and Unix systems are tar, tar.gz and tar.bz2 archives. Note that tar.gz and tar.bz2 archives are simply gzip-ped and bzip(2)-ped tar archives, respectively. Working with these files is made very simple through the use of GNU tar utility, which is included as part of the base packages in modern distros. In this article, I'll show you how to create and extract compressed archives using tar.

So, how is tar used? Lets start by looking at some of the common tar operations and options.

man tar # ©2007 linux.dsplabs.com.au

The following tar usage information is taken directly from the man pages.

Usage:
tar [options]

Common Operations:
-A, --catenate, --concatenate append tar files to an archive
-c, --create create a new archive
-r, --append append files to the end of an archive
-x, --extract, --get extract files from an archive

Common Options:
-f, --file [HOSTNAME:]F use archive file or device F
(default "-", meaning stdin/stdout)
-j, --bzip2 filter archive through bzip2,
use to decompress .bz2 files
-p, --preserve-permissions extract all protection information
-v, --verbose verbosely list files processed
-z, --gzip, --ungzip filter the archive through gzip,
use to decompress .gz files

Note that the GNU tar utility does not require a preceding minus for the single letter options (thanks to correct, see comments below).

Creating archives

To create a tar archive the c switch is used. To further encode it using gzip compression the z option is also added, or for bzip2 compression the j switch is used instead. Note that tar program pipes its output into gzip and bzip2 tools in order to create the tar.gz and tar.bz2 archives, respectively. OK, to compress a directory called dir into dir.tar, dir.tar.gz and dir.tar.bz2 archives, the following commands are used, respectively.

tar cf dir.tar dir/ # ©2007 linux.dsplabs.com.au
tar czf dir.tar.gz dir/ # ©2007 linux.dsplabs.com.au
tar cjf dir.tar.bz2 dir/ # ©2007 linux.dsplabs.com.au

In the above examples, the use of the f option specifies that the compressed version of the dir directory is to be placed in a corresponding archive file. On the other hand, if the f option was not given (and thus the archive name was also omitted), then the stdout, i.e. your terminal screen, would be used as the output instead. This is useful if you want to pipe the output of your tar command into another Linux tool. Anyhow, lets have a look at the resulting archives as well as the original directory using ls -la.

total 424
drwx------ 3 kamil kamil 4096 Nov 27 22:39 .
drwx------ 3 kamil kamil 4096 Nov 27 22:34 ..
drwx------ 2 kamil kamil 4096 Nov 27 22:36 dir
-rw------- 1 kamil kamil 276480 Nov 27 22:39 dir.tar
-rw------- 1 kamil kamil 83330 Nov 27 22:39 dir.tar.gz
-rw------- 1 kamil kamil 45927 Nov 27 22:39 dir.tar.bz2

Lets also have a look at the size of the original directory using du -sh dir # ©2007 linux.dsplabs.com.au.

280K dir

From the above shell output, you can see that by default tar only archives files without compressing them, while gzip and bzip2 filters achieve quite high compression. Note that bzip2 typically achieves better compression that gzip, although it might take longer time to do so. Also note, that in this case gzip and bzip2 filters achieve quite high compression ratios because the dir directory contains text files. If the directory contained already compressed files, say f.e. binary images compressed using the JPEG compression, then neither gzip nor bzip2 could do much more in terms of compression.

Extracting archives

Extracting archives is also very simple. Instead of the c switch the x is used and the archive name is given as the only other parameter. The commands for archive extraction shown below correspond to the archive creation commands given earlier.

tar xf dir.tar # ©2007 linux.dsplabs.com.au
tar xzf dir.tar.gz # ©2007 linux.dsplabs.com.au
tar xjf dir.tar.bz2 # ©2007 linux.dsplabs.com.au

Infact, in most cases, tar will figure out what archive type you are trying to extract (from their hex headers I suppose), so that the filter specifications are not really needed. Hence, the following still works fine.

tar xf dir.tar # ©2007 linux.dsplabs.com.au
tar xf dir.tar.gz # ©2007 linux.dsplabs.com.au
tar xf dir.tar.bz2 # ©2007 linux.dsplabs.com.au

However, if you explicitly specify the decoder, then tar will assume that that is the encoding of the given archive. If for whatever reason that is not the case, an error will occur.

The verbose mode

The v switch can be used to enable the verbose mode. This can be useful if you would like to see a list of files being compressed or extracted. For example, lets extract the dir.tar.gz archive, with verbose mode enabled, using the following command.

tar xvzf dir.tar.gz # ©2007 linux.dsplabs.com.au

The above command produces a list of inflated files as shown in the following output.

dir/
dir/NVIDIA_DRIVER_README.txt
dir/NVIDIA_LICENSE.txt
dir/readme.txt

Some common errors

Lets take a look at some common error examples. As described previously, specifying incorrect filter type, f.e. using the following commands

tar xjf dir.tar # ©2007 linux.dsplabs.com.au
tar xzf dir.tar.bz2 # ©2007 linux.dsplabs.com.au

results in the respective errors messages shown below.

bzip2: (stdin) is not a bzip2 file.
tar: Child returned status 2
tar: Error exit delayed from previous errors

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error exit delayed from previous errors

Also, if a file becomes damaged, say truncated, then obviously an error will occur. Lets simulate such a corruption. Lets keep only the initial 100 bytes of the dir.tar.bz2 archive by using the following command.

head -c100 dir.tar.bz2 > corrup.tar.bz2 # ©2007 linux.dsplabs.com.au

If we now try to extract this archive,

tar xjf corrupt.tar.bz2 # ©2007 linux.dsplabs.com.au

then the following error message will be produced.

bzip2: Compressed file ends unexpectedly;
perhaps it is corrupted? *Possible* reason follows.
bzip2: Inappropriate ioctl for device
Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

tar: Child returned status 2
tar: Error exit delayed from previous errors

Interestingly, we are told to check archives integrity, so lets do that.

bzip2 -tvv corrupt.tar.bz2 # ©2007 linux.dsplabs.com.au

The bzip2 utility tells us what we already know… we are missing part of the file.

corrupt.tar.bz2: [1: huff+mtf file ends unexpectedly

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

Lets try the second suggestion, the bzip2recover utility.

bzip2recover corrupt.tar.bz2 # ©2007 linux.dsplabs.com.au

Unfortunately, with only 100 bytes it is not possible to recover anything from the corrupted archive.

bzip2recover 1.0.3: extracts blocks from damaged .bz2 files.
bzip2recover: searching for block boundaries ...
block 1 runs from 80 to 800 (incomplete)
bzip2recover: sorry, I couldn't find any block boundaries.

Similarly, if we truncate the gzip archvie, then gzip will inform us of unexpected end of file.

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error exit delayed from previous errors

On the other hand, truncating a tar archive does not cause error messages during extraction process. Obviously, only the files still fully contained (an not truncated/corrupted) in such an archive will be extracted. Such tuncation/corruption errors often occur when extracting archives downloaded from the Internet.

Can I Set Up User Name Mapping in Windows Vista?

2009-12-19T20:32:00.000+02:00

Can I Set Up User Name Mapping in Windows Vista?

Unfortunately, no if you don't have a SFU 3.5 or W2K3 R2 machines on the network that is running User Name Mapping service. This has been a major disappointment with NFS deployments using Windows Vista.

However, there is a tiny little good news for you if you are looking for a way to enable Client for NFS in Windows Vista to be able to do something other than anonymous mounts. There is a registry tweak that you can modify to tweak Vista NFS client to use a pair of UNIX UID and GID while mounting and accessing NFS shares.

Here's how you can do it -

Start Registry Editor
Locate HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ClientForNFS\CurrentVersion\Default
Create two DWORD values namely AnonymousUid and AnonymousGid
Set these values to the UID and GID you would like this NFS client to use
Restart your Client for NFS service using the Microsoft Services for NFS MMC snap-in
Go ahead and mount shares on this machine and now the Client for NFS driver will send those values as the UID and GID to the NFS server. You can also use the mount command without any parameters to verify if these values are being used. You should see something like this -

C:\>mount

Local Remote Properties
-------------------------------------------------------------------------------
x: \\NFSSERVER\NFSSHARE UID=10012, GID=10011
rsize=8192, wsize=8192
mount=soft, timeout=1.6
retry=1, locking=yes
fileaccess=644, lang=ANSI
casesensitive=no
Try creating a file on the share and on the UNIX box, you can see that they bear the same UID and GID values as we have mentioned in the above DWORD values. If you use the ls command that comes with SUA, you will not see the correct ownership information because the client is still doing anonymous mounts and as you might have guessed already - we are just changing the UID and GID that the client is sending to the server. It is more like squashing from a client side instead of doing it from the server.

There's a security consideration as well - Every user on this machine will start sending these same NFS credentials so if you are going to have multiple users using this machine - think carefully about using this registry tweak. You also will not be able to use any secondary groups. I guess that can also be a consideration for many environments.

BTW, this works the same way on Windows 7 box ;)

Bandwidth Monitoring Tools for Ubuntu Users

2009-11-23T18:00:00.001+02:00

Bandwidth in computer networking refers to the data rate supported by a network connection or interface. One most commonly expresses bandwidth in terms of bits per second (bps). The term comes from the field of electrical engineering, where bandwidth represents the total distance or range between the highest and lowest signals on the communication channel (band).
Bandwidth represents the capacity of the connection. The greater the capacity, the more likely that greater performance will follow, though overall performance also depends on other factors, such as latency.

Here is the list of bandwidth monitoring tools for your network bandwidth

bmon
bwbar
bwm
bwm-ng
iftop
iperf
ipfm
speedometer
cbm
ibmonitor
pktstat
mactrack
MRTG
Cacti

Now we will see each tool separately

bmon

bmon is a portable bandwidth monitor and rate estimator running on various operating systems. It supports various input methods for different architectures.

Various output modes exist including an interactive curses interface, lightweight HTML output but also formatable ASCII output

Current Stable Version :- 2.1.0

Install bmon in Ubuntu

sudo apt-get install bmon

This will complete the installation.

If you want to open the application you need to enter the following command

bmon

you should see the following output

If i select the eth0 i should see all the traffic details

If you want to know more available options check man page

bwbar

This program will output a PNG and a text file that can be used in scripts or be included in web pages to show current bandwidth usage. The amount of total bandwidth can be customized. The PNG output appears as a bar graph showing maximum possible usage with the current inbound or outbound usage shown as a differently colored bar.

Current Stable Version :- 1.2.3

Install bwbar in ubuntu

sudo apt-get install bwbar

This will complete the installation

There is a /etc/default/bwbar file to configure before this package will work and here is my config

# Directory to put files into
DIR=/var/www
# Who to run as
RUNAS=www-data
# What are the options (eth0, scale of 1.5mbps)
OPTIONS=”eth0 1.5″

Now you need to chnage the permissions using the following command

sudo chown -R www-data:www-data /var/www/bwbar/

Start the bwbar using the following command

sudo /etc/init.d/bwbar start

You can also launch bwbar using the following command

bwbar eth0 100 -x 200 -y 7 -t 10 -p network.png -f network.txt -d /var/www/bwbar &

Now you need to go to http://serverip/bwbar you should see the bandwidth bar as follows

If you want to know about available option check man page

bwm

BandWidth Monitor This is a very tiny bandwidth monitor (not X11). Can monitor up to 16 interfaces in the in the same time, and shows totals too.

Current Stable Version :- 1.1.0

Install bwm ubuntu

sudo apt-get install bwm

This will complete the installation now

If you want to run the application use the following command

bwm

Output looks like below

If you want more details and available options check man page

bwm-ng

small and simple console-based bandwidth monitor.Bandwidth Monitor NG is a small and simple console-based live bandwidth monitor.

Current Stable Version :- 0.5

features

supports /proc/net/dev, netstat, getifaddr, sysctl, kstat and libstatgrab

unlimited number of interfaces supported

interfaces are added or removed dynamically from list

white-/blacklist of interfaces

output of KB/s, Kb/s, packets, errors, average, max and total sum

output in curses, plain console, CSV or HTML

configfile

Install bwm-ng in ubuntu

sudo apt-get install bwm-ng

This will complete the installation

If you want to run the application you need to use the following command

bwm-ng

Output looks like below

If you want more details and available options check man page

iftop

iftop does for network usage what top does for CPU usage. It listens to network traffic on a named interface and displays a table of current bandwidth usage by pairs of hosts.

Current Stable Version :- 0.17

Install iftop in ubuntu

sudo apt-get install iftop

This will complete the installation

If you want to run the application you need to use the following command

iftop

Output looks like below

If you want more details and available options check man page

ipfm

IP Flow Meter (IPFM) is a bandwidth analysis tool, that measures how much bandwidth specified hosts use on their Internet link.

Current Stable Version :- 0.11.5

Install ipfm in Ubuntu

sudo apt-get install ipfm

This will complete the installation

Now if you want to configure this for your network you need to copy example configuration file from /usr/share/doc/ipfm/examples if you want to check this file click here

once you configure this file you need to start the service using the following command

sudo /etc/init.d/ipfm start

If you want more details and available options check man page

Speedometer

Measure and display the rate of data across a network connection or data being stored in a file.

Current Stable Version :- 2.4

Install speedometer in ubuntu

Check which version of python is the default by running

python -V

Then issue the following commands as root to install speedometer (choose the correct Urwid package for your python version, ie. if python -V reports version 2.3.X then install python2.3-urwid)

apt-get install python2.4-urwid

Download the speedometer.py source file.

As user issue the following commands in the directory that you downloaded the source file

sudo cp speedometer.py /usr/local/bin/speedometer

sudo chown root: /usr/local/bin/speedometer

sudo chmod 755 /usr/local/bin/speedometer

Now you can run the speedometer application using the following

/usr/local/bin/speedometer

Now you should see the following screen output

Speedometer Usage

Usage: speedometer [options] tap [[-c] tap]

Available options

speedometer -h

Usage: speedometer [options] tap [[-c] tap]…

Monitor network traffic or speed/progress of a file transfer. At least one tap must be entered. -c starts a new column, otherwise taps are piled vertically.

Taps:

[-f] filename [size] display download speed [with progress bar]

-f must be used if directly following another

file tap without an expected size specified

-rx network-interface display bytes received on network-interface

-tx network-interface display bytes transmitted on network-interface

Options:

-i interval-in-seconds eg. “5″ or “0.25″ default: “1″

-p use plain-text display (one tap only)

-b use old blocky display instead of smoothed display even when UTF-8 encoding is detected

-z report zero size on files that don’t exist instead of waiting for them to be created

Usage Examples

How long it will take for my 38MB transfer to finish?

speedometer favorite_episode.rm $((38*1024*1024))

How quickly is another transfer going?

speedometer dl/big.avi

How fast is this LAN?

$ cat /dev/zero | nc -l -p 12345

$ nc host-a 12345 > /dev/null

$ speedometer -rx eth0

How fast is the upstream on this ADSL line?

speedometer -tx ppp0

How fast can I write data to my filesystem? (with at least 1GB free)

dd bs=1000000 count=1000 if=/dev/zero of=big_nothing &
speedometer big_nothing

cbm

cbm — the Color Bandwidth Meter — displays the current traffic on all network devices.

Current Stable Version :- 0.1-1

Install cbm in ubuntu

First you need to download the .deb package from here

once you have the .deb package you need to install using the following comamnd

sudo dpkg -i cbm_0.1-1_i386.deb

this will complete the installation now if you want to use the appltcation use the following command

cbm

Output looks like below

pktstat

pktstat listens to the network and shows the bandwidth being consumed by packets of various kinds in realtime. It understands some protocols (including FTP,HTTP, and X11) and adds a descriptive name next to the entry (e.g., ‘RETR cd8.iso’, ‘GET http://slashdot.org/’ or ‘xclock -fg blue’).

Current Stable Version :- 1.8.3

Install pktstat in Ubuntu

First you need to download .rpm package from here once you have the .rpm package you need to convert this .rpm file to .deb file using alien

Install alien

sudo apt-get install alien

Now you need to use the follwoing command to convert .rpm to .deb

sudo alien -k pktstat-1.7.2q-0.i386.rpm

Now you should be having pktstat_1.7.2q-0_i386.deb package

Install pktstat in Ubuntu

sudo dpkg -i pktstat_1.7.2q-0_i386.deb

This will complete the installation now you can open the application using the following command

pktstat

output looks like below

If you want more available options for pktstat check man page

ibmonitor

ibmonitor is an interactive linux console application which shows bandwidth consumed and total data transferred on all interfaces.

Current Stable Version :- 1.4

Its main features are:

Shows received, transmitted and total bandwidth of each interface

Calculates and displays the combined value of all interfaces

Displays total data transferred per interface in KB/MB/GB

Values can be displayed in Kbits/sec(Kbps) and/or KBytes/sec(KBps)

Can show maximum bandwidth consumed on each interface since start of utility

Can show average bandwidth consumption on each interface since start of utility

The output with all features (max, avg and display in Kbps and KBps) easily fits on a 80×24 console or xterm

Can interactively change its output display format depending on key pressed by user.

Install ibmonitor in Ubuntu

First you need to download the latest version from here

wget http://ovh.dl.sourceforge.net/sourceforge/ibmonitor/ibmonitor-1.4.tar.gz

Now you have ibmonitor-1.4.tar.gz

Extract this file using the following commands

tar xvfz ibmonitor-1.4.tar.gz

cd ibmonitor

If you want to run the application use the following command

Once you are in ibmonitor folder use

./ibmonitor

Output looks like the following screen

iperf

While tools to measure network performance, such as ttcp, exist, most are very old and have confusing options. Iperf was developed as a modern alternative for measuring TCP and UDP bandwidth performance.

Iperf is a tool to measure maximum TCP bandwidth, allowing the tuning of various parameters and UDP characteristics. Iperf reports bandwidth, delay jitter, datagram loss.

Current Stable Version :- 2.0.2

Install iperf in ubuntu

sudo apt-get install iperf

iperf Syntax

iperf [-s|-c host] [options]

Example

iperf -c server address -F file-name

iperf -c server address -I

The -F option is for file input.

The -I option is for input from stdin.

If you want more details and available options check man page

tcptrack

tcptrack is a sniffer which displays information about TCP connections it sees on a network interface. It passively watches for connections on the network interface, keeps track of their state and displays a list of connections in a manner similar to the unix ‘top’ command. It displays source and destination addresses and ports, connection state, idle time, and bandwidth usage.

Current Stable Version :- 1.1.5

Install tcptrack in Ubuntu

sudo apt-get install tcptrack

this will complete the installation

tcptrack Syntax

tcptrack [-dfhvp] [-r ] -i []

Examples

tcptrack requires only one parameter to run: the -i flag followed by an interface name that you want tcptrack to monitor. This is the most basic way to run tcptrack

tcptrack -i eth0

tcptrack can also take a pcap filter expression as an argument. The format of this filter expression is the same as that of tcpdump and other
libpcap-based sniffers. The following example will only show connections from host 10.45.165.2

tcptrack -i eth0 src or dst 10.45.165.2

The next example will only show web traffic (ie, traffic on port 80)

tcptrack -i eth0 port 80

The following output screen will show you more details

MRTG

The Multi Router Traffic Grapher or just simply MRTG is free software for monitoring the traffic load on network links. It allows the user to see traffic load on a network over time in graphical form.

Current Stable Version :- 2.15.0

Project Homepage

http://oss.oetiker.ch/mrtg/

Cacti

Cacti is a complete network graphing solution designed to harness the power of RRDTool’s data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive,easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices.

Current Stable Version :- 0.8.6i

Project Homepage

http://www.cacti.net

Regarding MRTG and Cacti i am going write detailed articles in future

Zend Performance

2009-11-06T14:49:00.001+02:00

Let me first start off by saying that the Zend Framework has been very good to us.

It enabled us to build a kick-ass application in a relatively short amount of time. On top of that, we followed the conventions from Zend and PEAR and essentially have a very maintainable piece of software which I don't hate looking at every day (which is as one can imagine, a huge plus).

The other day our servers were overwhelmed with the rising traffic and I started profiling my application through Xdebug. Initially I tried to use Zend Studio and the Zend_Debugger but Zend doesn't like my (awesome) operating system (FreeBSD) and only provides Linux and Windows extensions. Xdebug, while being free and awesome in general, doesn't know this prejudice. :-)

On this project we currently run with 100,000 visitors per day on average, our peak is Sunday night where we get a ton more traffic than usually. We run the latest PHP (5.2.6 at this time), etc.. The software comes from FreeBSD ports, there are no magic secret patches. I'm picky about the modules I compile and load but the list is far from optimized.

To our defense, we just relaunched over summer and since we are a team of four total and only two of those four people write code. Since we started off slow with 60,000-80,000 visitors per day since summer, we never really had a chance or need to optimize and tried to avoid all premature optimization.

We currently use 50-some Zend-classes. I wish I could provide a better number, but as you may know, the Zend Framework is only bundled as a whole and figuring out which classes are all in the mix is tricky. So the 50 is an estimation based on grepping through our own class code mostly.

On the server we run Apache 1.3. Currently we have a total of four webservers (two older (dual core, 6 GB RAM, slow disks (7.5k rpm)), two newer models (eight cores and 6 GB RAM, faster disks (15k rpm)). The backend consists of two powerful workhouses with eight cores, more RAM (than the frontends) and a lot of disk (at 12k rpm each).

Prior to starting the quest for performance one of our older servers was able to handle ten (10) requests/second at peaks, now we are at 42 requests/second. (Give or take a few.) In regard to page loading time, we went with a few optimizations from 340 ms to 76 ms in no time (all figures according to Xdebug). So I feel like we are right on the right track to Getting rich with PHP. (Where's my Lexus at? :-))

We benchmarked using Apache Bench (at moderate ab -n 1000 -c 100 http://url) and Siege, which are both really awesome tools and provide you with an instant DoS attack on your servers. I might add that you are better off running those tools from "localhost" vs. remote as you might trigger your providers IDS/snort/DoS protection otherwise.

Here are a few things, that helped us. Suggestions are in no particular order and I should add that whatever is applicable for my suitation, doesn't have to work for you. Also, my number game could be off and if you have suggestions on how to improve, please comment or drop me a line.

1) APC
It cannot be stressed enough. Please run APC. Please take into account to adjust the default settings, also check apc.slam_defense, apc.write_lock and apc.stat.

We had APC before, but I felt like I needed to mention it on the list of things.

Also, apc_fetch, apc_store etc. are great ways to add little caches throughout your application.

And they almost require zero time to implement. I may suggest you use apc_fetch/apc_store directly vs. wrapping the Zend_Cache layer around it which provides (IMHO) little added value and benefit but just adds more class code around the obvious.

2) Adjust PHP's realpath.cache setting and .ttl
This helps, somewhat.

3) Get rid off require_once, use __autoload (and the Zend_Loader)

This might be a hassle when during development, because require_once evaluates each include, thus letting you know if it finds a parse error, and also where.

With include_once (which Zend_Loader essentially uses), it's a bit tricky at times. A good idea here would be a phing task (or some other script) which strips out or replaces require_once when you deploy your application to production.

Removing require_once in favour of __autoload shows one of the biggest performance improvements in my entire application - I shaved off roughly 220 milliseconds by removing about 15 (or so) calls to require_once in my bootstrap.php file. And that's with APC enabled, and a decent sized realpath.cache (and .ttl).

Beyond weird coding conventions (I shall bitch about those in another blog post), require_once is also the number one performance killer from the entire Zend_* code base. The before/after is amazing. Without any of those enhancements from the list just by stripping out require_once from our ZendFramework "install", we went from 9-10 requests/second to 27 requests/second.

Use the following shellscript to strip them:

grep -rl require_once . | grep -v svn |grep -v Loader | xargs perl -pi~ -e 's/require_once/#require_once/'

3) Zend_Loader
I know, I just recommended using the Zend_Loader but with no offense, the Zend_Loader sucks is not so great when it comes to general performance. Obviously I did not write it and really no offense meant, but it does some really weird stuff on the inside which I am not sure what the use-case is. But I am sure there is one. ;-)

In order to preserve the API, I extended Zend_Loader and started overwriting functions such as Zend_Loader::_securityCheck(), which runs a regular expression on the name of each file you feed to __autoload/Zend_Loader.

On top of that I switched to usig the Zend_Loader only for models and controllers. But not for Zend_ and Company_-classes. Since Zend (and we) essentially follow the great PEAR coding standard in regard to one class per file and a very explicit naming scheme, all you have to do in your __autoload is the following:

function __autoload($className) {
include_once $className = str_replace('_', '/', $className) . '.php';
}

Now, that would be the bare minimum and our loader looks slightly more complicated but I haven't stopped there and we are still in the process of "dumbing" it down even further, but so far it saved us between five and 15 ms per page.

4) Cache DB results, avoid queries!

Those tricky, tricky DB queries.

Even though our DB backends idle mostly even when we get beat with traffic, there's a few things to keep in mind.

One of them is - DB queries are really expensive. And by queries I am not talking about the "SELECT * FROM foo"-part, but rather about opening a connection to another server, sending the query, receiving it and so on. Let alone by caching one of those, we roughly gained another 20 ms on the frontpage. And it's not a very complex query either.

I remember looking puzzled when ahem... I was presented with the code that pulls a status message on each request to the homepage but I had forgotten about this already and just noticed it again when it popped up in xDebug with a notable amound of milliseconds.

5) Zend_Db_Table

Zend_Db_Table is very easy to use, in fact most of our models wrap around a couple tables and that's why we got a bunch of them. Now what I did not realize (but thanks to JamesG@#zftalk now I do), is that the meta data the class uses to provide all those nifty interfaces is generated on each request. That's a DESCRIBE TABLE in the background, which is pure overhead.

Zend_Db_Table_Abstract::setDefaultMetadataCache() to the rescue.

5) Apache

5a)
I sometimes hate Apache, but I also can't live without it.

Over the past years, I have tried all sorts of things in the webserver market - Lighttpd and nginx with php-cgi (fastcgi) seem to be no fun. A commercial solution such as Resin or Zeus has never been an option either.

I've always come back to Apache (1.3) for the simple fact that Apache and PHP are really so tightly integrated that nothing ever will go wrong.

Remember that guy Nik who claimed that Apache/PHP sometimes fail and deliver the sourcecode to the browser (because Facebook obviously failed to configure Apache)? Well, that doesn't happen - ever. The only problem with Apache is that Apache and client(browser)-communication is a bitch.

Nginx to the rescue! Fast install, easy to configure (don't let the Russian FAQ scare you, the nginx.conf-dist will teach you all you need!) - just chain your Apache to localhost:8080 and let Nginx proxy all requests to it and your Apaches move from "lockf" status, to "run" and "accept" always.

Whenever Apache receives a request of a slower client, it will have to wait until the slower client is done reading all of the response. While waiting, your 30 MB Apache sits there unable to do anything else. Which nginx in the mix, the Apache sends the response as fast as it can to nginx, thus having more time to take care of what it's supposed to do for you - PHP.

Judging from my poor benchmarks, nginx adds to the number of requests by factor six or seven (6 or 7). It's amazying and I never expected it to have such a great impact. It also doesn't eat away on resources, so beware of the Russians! :-)

Take all "optimizations" into account, Apache 1.3, proxied by Nginx can now handle over 3000 requests/second (ab -n 10000 -c 1000 http://url).

5b)
The obvious quirks, for example check out your default Apache install and unload all the modules and extensions you never use anyway.

For example, we don't have any of those HTTP authentication boxes ever. So why do we need *_auth_* modules. Then, we don't use a user_dir, why load mod_userdir, our Apache does not log - why load mod_config_log, or my most favorite: mod_status.

Make sure mod_status is really disabled because otherwise that's one very, very expensive operation you got right there, with each request.

A good idea is to check top, unload, and look again:

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
42601 root 1 96 0 7092K 2324K CPU1 1 0:00 1.00% top
36242 www 1 20 0 114M 33304K lockf 1 0:22 0.29% httpd
38251 www 1 20 0 114M 32184K lockf 0 0:08 0.29% httpd
42579 www 1 20 0 114M 28016K lockf 1 0:02 0.29% httpd
37975 www 1 20 0 115M 34688K lockf 1 0:18 0.24% httpd
36344 www 1 20 0 115M 34036K lockf 1 0:18 0.24% httpd

Take into account (thanks, Jan), that the size column, is not the real size. It's like the theoretical size including whatever the Apache could use if it had to, but you want to look at RES (resident) instead. Because that's what's in the memory right now.

Another smart move is to put all the rules from .htaccess into your server configuration because otherwise Apache searches for various (!) .htaccess files with each requests and tries to evaluate the rules you have in there.

Imagine this:

/htdocs/foo/bar/index.html

In this request Apache will look for .htaccess in the following directories:
/htdocs
/htdocs/foo
/htdocs/foo/bar

Turn it off (AllowOverride none) instead and move on because when you move all your directives into httpd.conf (or similar) at least they get evaluated once when the server process starts up.
From a deployment perspective it's nicer to have .htaccess because all you need to do is re-deploy one file vs. editing a server config and restarting a server, but this really pays off. With certain APC settings, you will need to restart the server anyway, also, "No pain, no gain!".

REPLACE in PostgreSQL

2009-10-29T10:37:00.002+02:00

I've recently made the switch from MySQL to PostgreSQL for some of my more complicated database projects, such as the ad matching discussed in my last post. PostgreSQL has a much richer feature set (although MySQL 5 is catching up). One feature I couldn't find in PostgreSQL however was the REPLACE extension offered by MySQL where rows that occur in duplicated primary keys replace the existing rows instead of give errors. Fortunately, PostgreSQL's advanced rules system allows the creation of an equivalent feature.

For the sake of this example we'll use a very simple table called 'map' with a two fields: 'key' and 'value'. Not surprisingly, key is the primary key.

Let's further assume that every insert into this table should actually be a replace. Then all we need is this simple statement and we're done:

-- Merge rule
CREATE OR REPLACE RULE merge_btv_lookup_weather_locations AS
ON INSERT TO btv.lookup_weather_locations
WHERE (EXISTS ( SELECT 1
FROM btv.lookup_weather_locations
WHERE lookup_weather_locations.code = new.code)) DO INSTEAD UPDATE btv.lookup_weather_locations SET name = new.name, type = new.type, active = new.active
WHERE lookup_weather_locations.code = new.code;

That's it! So how does it work? First, we create a rule to be processed whenever a row is inserted into the map table. Then, we use the subquery EXISTS(SELECT 1 FROM map WHERE key=NEW.key) to test for a row on the same primary key. If there isn't a row, the query proceeds as a normal INSERT. But if there is a row, the query is converted into an UPDATE that sets the value instead of trying to insert it. The 'NEW.var' means whatever values we were trying to INSERT.

This is why I'm starting to become a PostgreSQL fan.

How to convert dmg file to an iso image

2009-10-11T09:21:00.000+03:00

I found this out a while ago when looking for info on reading .DMG files on Windows or Linux boxes. I found out it was not possible, and I wasn't too happy. You see, my iMac has no CD-R drive, just a CD-ROM. This tip creates ISO images from DMG images, so they can be burned elsewhere. To convert the file to an ISO image, type the following command at your terminal window:

hdiutil convert /path/to/filename.dmg -format UDTO -o /path/to/savefile.iso

Replace /path/to/filename.dmg with the path and name of the existing .DMG file, and replace /path/to/savefile.iso with the desired path and name for the converted image.

This then creates an ISO image burnable in Nero on Windows, or pretty much anything on Windows that will burn ISOs and same with Linux. I just converted a DMG image as a test, and it took a while -- it only converted at about 1MByte per secoond, but I only have a 333Mhz imac G3, so speed wise, it may be good.

2009-08-19T12:51:00.001+03:00

Even if you use OpenOffice, you might still want all the Microsoft TrueType fonts so that documents created using Word or PowerPoint look as they were supposed to when you open them with OpenOffice. Also, with the Microsoft Fonts installed we browsing will be better since the pages will look as the designer originally intended them to. Most webpages are designed with Microsoft fonts in mind. The stylesheet specify these fonts. On Linux, when these specified fonts are not available on your computer, they are replaced with generic equivalents. With these fonts installed, you will see the page as it was designed. To install the fonts, all you need to do in Ubuntu is to install the msttcorefonts package. Instructions for installation are given below.

The Truetype Microsoft fonts provided by the package include:

Andale Mono
Arial Black
Arial (Bold, Italic, Bold Italic)
Comic Sans MS (Bold)
Courier New (Bold, Italic, Bold Italic)
Georgia (Bold, Italic, Bold Italic)
Impact
Times New Roman (Bold, Italic, Bold Italic)
Trebuchet (Bold, Italic, Bold Italic)
Verdana (Bold, Italic, Bold Italic)
Webdings

Installing Microsoft Truetype fonts on Ubuntu

You can install the MS core fonts by installing the msttcorefonts package. To do this, enable the “Universe” component of the repositories. This is done by default in Feisty. After you do that, use the following command from the command line:

$sudo apt-get install msttcorefonts

This will give you the core fonts, but if there are other TrueType fonts that you want installed, it is as easy as copying the font files to the ~/.fonts/ directory.

After installing new fonts, you will have to log out and log in again to be able to see and use the new fonts. If you want to avoid this, you can regenerate the fonts cache by issuing the following command:
$sudo fc-cache -fv

Enable and Disable Ubuntu Root Password

2009-06-14T15:31:00.001+03:00

Ubuntu is one of the few Linux distributions out there that will not enable the root account.If you want to do something with root permission on the console you have to type sudo before the command.

sudo” means superuser do. “sudo” will prompt for “Password:”. Please specify user password

As you have noticed during the Ubuntu installation there was no question about the root password, as you might have been used to see during other Linux distribution installation process.Because of this your root accout is inactive.

If you want to enable root account (which is not recommended) enter the following command.

$sudo passwd root

This will prompt for a new root password and once you confirm it, you can start using the root account to login.

If you want to disable root account in ubuntu you need to lock the root account by using the following command

$sudo passwd -l root

If you want to work on a root console you’d better use the following command

$sudo -i

2009-06-07T10:19:00.001+03:00

Как да си изключа swap partion-а, за да проверя дали няма да имам подбрение в performance-a. Тази стъпка се препоръчва на машини с повечко RAM, които се ползват desktop или тестови машини, в никакъв случай не препоръчвам тази стъпка за машини в production. Идеята е да изстискаме малко performance, но машината трябва да се наблюдава, за да няма странни проблеми.

How to grep in files by specific extension

2009-06-04T22:28:00.003+03:00

For example lets search for "*.php" files

We should have something like this:

find ./ -name "*.php"

Now let grep for specific string "$GLOBALS" in a specific php file - test.php

grep -n "$GLOBALS" test.php

Now let combine what we learn

find ./ -name "*.php" -type f -exec grep -H -n "$GLOBALS" {} \;

the above code search the current directory for php files and search in them "$GLOBALS"

-H -n means show me filename and line number.

This should work.

For best results use GUI editor with search in files :-)

For more info

man grep

How to remove '*.svn' files from a repository copy

2009-06-04T22:25:00.001+03:00

find . -name '*.svn' | while read i; do rm -r -d -f "$i"; done