irqtune changes the IRQ priority of devices to allow devices that require high priority and fast service (e.g. serial ports, modems) to have it.
With irqtune, a 3X speedup of serial/modem throughput is possible.
You probably need irqtune, if you are experiencing any of the following:
When the PC boots Linux, the timer is given, by default, the highest IRQ priority in the system (it's IRQ 0 and thus, priority 0). On a standard configuration, the serial ports are priority 11 and 12!!! This means that 10 other devices have higher priority.
When multiple devices are in contention to interrupt the CPU, their priority decides which interrupts will occur in what order.
After an arbitrary period of having interrupts disabled (e.g after a cli), at the point where they're reenabled (sti). This can happen in several places:
In the default configuration, the serial ISR will usually lose as it's priority 11.
irqtune gives priority 0 to whatever device we specify. If we specify a serial device, irqtune guarantees that the serial ISR gets control whenever a contention occurs.
Serial devices are somewhat unique. Even though they have one of the slowest data rates (relative to a disk), they are the largest consumer of interrupts and are extremely sensitive to interrupt latency (the time from when a device yanks the IRQ line until its ISR is executed).
Unlikely doesn't mean never. There are places were the contention period must occur, no matter how we program the CPU.
Variations in CPU speed, RAM size, RAM speed, cache size, disk speed, disk rotational position, number and type of other devices, system workload, etc, etc etc, all contribute to variations of order and timing of internal OS events.
Unlikely on one system may mean 1/1000th % chance. On another system, it may mean happens 50 times/second. Beyond a certain point, it all comes down to measurement.
Tight, accurate, repeatable measurement is the key to system tuning. If we can't measure it, we can't tune it. Once we can measure these things, we can then try various combinations until we achieve our desired results. It can often be pointless to guess at performance.
In fact, the interrupt disable windows themselves are used to prevent data corruption caused by various parts of the kernel that try to update a data structure simultaneously and get corrupted. Many of these windows are put there in the unlikely case that such corruption would occur. In this context, unlikely is never considered a reason to forego interrupt locking. Eliminating a necessary interrupt lock window could result in kernel panics, RAM corruption, disk data corruption, etc.
Not really.
In actual practice, most devices don't even notice the difference. Most other devices (e.g. disks, tape, ethernet) are DMA devices. The DMA does most of the work, thus greatly reducing their need for interrupts. If the device allows a request queue, it may function autonomously on several requests, producing only one interrupt for the entire batch.
Furthermore, serial interrupt services are, themselves, very fast. They slam their data as quickly as possible and get out ASAP. No fancy calculations, just the minimum, mindless data transfer. Almost everything else is handled later, in the bottom-half with interrupts enabled. In fact, a serial ISR may have to re-interrupt it's own bottom-half several times.
Those devices that do experience some slight slowdown are more likely to have long interrupt disable windows themselves. Having several smaller cli/sti windows is much better than one large cli/sti window--It's just harder to program.
Well, actually a prioritized system behaves like a balanced system--most of the time. This occurs when all devices have short interrupt lockout windows and short ISR execution times. The priority mechanism is like a safety valve--it only really matters when some device or combination of devices has held interrupts locked for an extended period of time.
When a disk ISR gets delayed, that's all that happens, a slight delay--disks and tapes are not real-time devices. When a serial ISR gets delayed, data is destroyed--serial devices are real-time devices that dictate the cadence of the entire system.
It may sound cruel, but you just can't be fair.
And speaking of sound, if the sound card gets delayed, we hear an annoying pop right in the middle of our favorite piece of music.
These are real-time devices that can not tolerate excessive delays. If we must defer some less time critical devices to meet the minimum real-time criteria of devices, then that's a bargain.
Ultimately, it's a bit of a compromise. Which is better:
No. It's actually an old idea. I've been doing device drivers since 1977 and Unix kernel work since 1981. I've personally written 8 serial drivers have used this many times commercially. Giving the serial device the highest priority is actually standard practice in many systems. With a 4Mhz CPU, these problems used to occur at 1200 baud :-)
cd /usr/local
tar zxvf irqtune.tgz
Note: If tar's
z
option has problems:
gzip -d < irqtune.tgz | tar xvf -
cd /usr/local/irqtune
make install
/sbin/irqtune
/sbin/irqtune_mod.o
/sbin/irqtune_npr.o
As a note to purists, the installation directory is completely arbitrary. /sbin is short, sweet, and easy to type.
The standard convention would be to install irqtune as /sbin/irqtune
and the .o files as, say, /usr/lib/irqtune/*.o.
irqtune uses a slightly different convention. It wants all installed
files to be in the same directory.
This works fine because argv[0] will
point to the .o files. This also allows:
If placing .o files in /sbin is deemed to be an
anathema, we have two options:
make SBIN=/whatever install
make INSTALL=sh install
Yes. If the shell/stub installation method is used:
irqtune will work even if we don't have the kernel source loaded. It uses insmod to load the patch, invoke it, and then unload it. The IRQ priority changes will last so long as the kernel is booted.
irqtune takes two arguments optional arguments:
usage: irqtune [options] [master] [slave]
version: 0.6
arguments:
master -- high priority IRQ on PIC master (DEFAULT: 3)
slave -- high priority IRQ on PIC slave (DEFAULT: 14)
general options:
-h -- display help
-v -- display irqtune version
-o -- reset to original values (0/8)
priority table options:
-q -- suppress priority table printing
-s -- sort table by priority
-x -- show inactive devices in table
error control options:
-e -- show full errors
-f -- force loading even if probe detects errors
-u -- force module unload
-V -- insmod verbose mode
-w -- treat warnings as errors
documentation options:
-F<file> -- use <file> instead of /proc/interrupts
-i -- install program
-L<directory> -- directory to search for insmod
-n -- nogo mode (just show what would happen)
Yes. Just add a /sbin/irqtune line to this file and we're in business. We may also issue another irqtune command at any time.
We can't.
Due to a limitation of the PC interrupt hardware, it is not possible to read back values set previously.
irqtune will attempt to place a message in the system log (/usr/spool/syslog/syslog) when it changes the configuration. Examining this log file, or simply invoking irqtune again, may be the best ways to work around the hardware limitations.
Note: Some users have tried to use the "-n" option, thinking it will act as a "query" mode. This option is used, primarily, to generate examples in this document and will not have the desired effect.
NOTE: For brevity, we've combined the non-sorted and sorted output in
these examples.
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P00: 8578913 timer I00/P00: 8578913 timer
I01/P01: 109547 keyboard I01/P01: 109547 keyboard
I02/P02: 0 + cascade I02/P02: 0 + cascade
I03/P10: 86470 + serial I11/P05: 197648 + sermux
I04/P11: 197648 + serial I12/P06: 17968 + eth
I11/P05: 197648 + sermux I13/P07: 1 math error
I12/P06: 17968 + eth I14/P08: 93123 + Ux4F
I13/P07: 1 math error I03/P10: 86470 + serial
I14/P08: 93123 + Ux4F I04/P11: 197648 + serial
NOTE: /proc/interrupts, and therefore irqtune, only reports on active devices. So to scope out the serial IRQ's, ideally, you'd have X Windows up with your serial mouse and be connected via PPP to the net.
The leftmost number is the IRQ number. The next number is the priority.
The rightmost column is the internal
device name (not to be confused with /dev names). In the above case, the two
serial ports are on IRQ 3 and IRQ 4. Just use the lower number, in this
case 3:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + serial
I01/P06: 109547 keyboard I04/P01: 197648 + serial
I02/P07: 0 + cascade I00/P05: 8578913 timer
I03/P00: 86470 + serial I01/P06: 109547 keyboard
I04/P01: 197648 + serial I02/P07: 0 + cascade
I11/P12: 197648 + sermux I14/P07: 93123 + Ux4F
I12/P13: 17968 + eth I11/P12: 197648 + sermux
I13/P14: 1 math error I12/P13: 17968 + eth
I14/P07: 93123 + Ux4F I13/P14: 1 math error
Glad you asked. There are actually two interrupt controllers, a master and a slave. The slave is cascaded to the master via its IRQ 2. The master controls IRQ's 0-7 and the slave controls IRQ's 8-15.
You actually may select two high IRQ priorities, one for the master and one for the slave. irqtune defaults the slave to IRQ 14, which is normally the disk controller.
In fact, cascade is sort of a "zero width" device as it does not contribute to interrupt latency. Setting the cascade to top priority on the master has an interesting effect which we'll see shortly.
In this case, we might want to use:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + serial
I01/P06: 109547 keyboard I04/P01: 197648 + serial
I02/P07: 0 + cascade I00/P05: 8578913 timer
I03/P00: 86470 + serial I01/P06: 109547 keyboard
I04/P01: 197648 + serial I02/P07: 0 + cascade
I11/P14: 197648 + sermux I12/P07: 17968 + eth
I12/P07: 17968 + eth I13/P08: 1 math error
I13/P08: 1 math error I14/P09: 93123 + Ux4F
I14/P09: 93123 + Ux4F I11/P14: 197648 + sermux
This is a bit tricky because now we've got a serial device on the slave controller. It would be much better to put all serial cards on the master controller. Things would stay much simpler.
In this case we would want to use:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P13: 8578913 timer I02/P00: 0 + cascade
I01/P14: 109547 keyboard I11/P00: 197648 + sermux
I02/P00: 0 + cascade I12/P01: 17968 + eth
I03/P08: 86470 + serial I13/P02: 1 math error
I04/P09: 197648 + serial I14/P03: 93123 + Ux4F
I11/P00: 197648 + sermux I03/P08: 86470 + serial
I12/P01: 17968 + eth I04/P09: 197648 + serial
I13/P02: 1 math error I00/P13: 8578913 timer
I14/P03: 93123 + Ux4F I01/P14: 109547 keyboard
Yes, this is shorthand way of saying 2 11. You can make a slave device top priority, but we get no options for the master IRQ. It will always be 2, the cascade device. Remember, the cascade device contributes no latency delay by itself.
Well, we boosted the priority of the serial multiplexer at the expense of the regular serial ports. The only way to allow all serial ports equally high priority is to group them on consecutive IRQ's and set the high priority for the lowest of those IRQ's.
We can't.
We're limited by the architecture of the PC and its interrupt controllers.
We must change the IRQ of a device by physical restrapping--we can't
do it by reprogramming the priority alone.
We'll go back to the earlier example. We'll wave a magic wand and
Poof!--assume we just restrapped the serial multiplexer to IRQ 5:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + serial
I01/P06: 109547 keyboard I04/P01: 197648 + serial
I02/P07: 0 + cascade I05/P02: 197648 + sermux
I03/P00: 86470 + serial I00/P05: 8578913 timer
I04/P01: 197648 + serial I01/P06: 109547 keyboard
I05/P02: 197648 + sermux I02/P07: 0 + cascade
I12/P13: 17968 + eth I14/P07: 93123 + Ux4F
I13/P14: 1 math error I12/P13: 17968 + eth
I14/P07: 93123 + Ux4F I13/P14: 1 math error
Sure. Waving the wand again, we restrap the ethernet card to IRQ 6.
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + serial
I01/P06: 109547 keyboard I04/P01: 197648 + serial
I02/P07: 0 + cascade I05/P02: 197648 + sermux
I03/P00: 86470 + serial I06/P03: 17968 + eth
I04/P01: 197648 + serial I00/P05: 8578913 timer
I05/P02: 197648 + sermux I01/P06: 109547 keyboard
I06/P03: 17968 + eth I02/P07: 0 + cascade
I13/P14: 1 math error I14/P07: 93123 + Ux4F
I14/P07: 93123 + Ux4F I13/P14: 1 math error
Exactly.
Different systems may have highly different criteria for what is optimum. It is, ultimately, a choice that each system administrator must make based upon the specific requirements for the particular system in question. We can only provide tools to do the job, but the final choice is ultimately decided on a case-by-case basis. There is no one size fits all solution.
irqtune makes every attempt to load its kernel module. irqtune probes the kernel, /proc/ksyms, and insmod. It attempts to detect, correct, or work around any difficulties. If the problems are truly severe, irqtune will report this also. Normally, irqtune probes silently, only reporting the results of the local system configuration if there's an non-recoverable error.
irqtune: version is 0.6
irqtune: kernel version 2.0.30
probe: irqtune must be invoked via the full path -- OK
probe: /sbin in $PATH -- YES
probe: insmod found in $PATH (/sbin) -- OK
probe: insmod simple execution -- OK
probe: insmod has version (2.0.0) -- YES
probe: rmmod found in insmod directory -- OK
probe: insmod version supports command line options -- OK
probe: insmod version (2.0.0) compatible with kernel version (2.0.30) -- OK
probe: insmod version should be 2.1.34 (or better) -- WARNING
probe: insmod and kernel compatible with CONFIG_MODVERSIONS -- OK
probe: irqtune_mod loading will be tried -- OK
probe: kernel version irqtune built under (1.0.0) matches current system -- NO
probe: kernel IRQ handling is compatible -- OK
probe: kernel has module support (CONFIG_MODULES) -- OK
probe: kernel has symbols -- OK
probe: kernel is using versions (CONFIG_MODVERSIONS) -- NO
probe: kernel symbols are checksummed (CONFIG_MODVERSIONS) -- NO
probe: kernel has /proc/interrupts -- OK
irqtune: simulating IRQ priority of 3/14
I00/P05: 8578913 timer
I01/P06: 109547 keyboard
I02/P07: 0 + cascade
I03/P00: 86470 + serial
I04/P01: 197648 + serial
I11/P12: 197648 + sermux
I12/P13: 17968 + eth
I13/P14: 1 math error
I14/P07: 93123 + Ux4F
irqtune: complete
No matter what happens with probes or loads, irqtune will report the final completion status as its last line: complete or error.
This does not matter. irqtune is 99.44% kernel revision independent. It is almost never necessary to rebuild the prebuilt modules. If irqtune fails to load the modules, consider everything in this section carefully before rebuilding irqtune.
Notes to programmers:
Rebuilding the binaries before exploring the other options is a lot like Vonnegut's OFF switch--it's comforting but not connected to anything :-)
When irqtune was first released, some experimental changes were made to the kernel to solve the IRQ priority problem by use of a round-robin, balanced priority system that was incompatible with irqtune. These changes were ultimately removed but irqtune will not work with kernel revisions 2.0.15 to 2.0.18. irqtune will detect this condition and report an error. See the Where can I find additional documentation or downloads? section.
irqtune first trys to load irqtune_mod.o and falls back to irqtune_npr.o if it detects a load error in insmod.
Some versions of insmod have severe difficulty loading modules when the
kernel is using MODVERSIONS. There is a known bug in insmod:
Generally, we should not use an older insmod with a newer kernel (e.g. using a 2.0.X insmod on a 2.1.X kernel). irqtune will detect and report this.
If insmod still has difficulties, we may want to upgrade it to 2.1.34 (or better). Newer versions of insmod are guaranteed to be backward compatible to older kernels. This will increase the probability that the irqtune_mod.o will load and irqtune will not have to fallback to irqtune_npr.o. Note: insmod is actually part of the modutils package. modutils under 2.0.X is called modules. See the Where can I find additional documentation or downloads? section.
The only kernel symbol that irqtune's kernel module (irqtune_mod.o) uses is printk (to print a confirmation message to syslog). The irqtune_npr.o module is exactly the same as irqtune_mod.o except that it does not use printk. Since irqtune pre-checks all parameters before attempting to load the kernel module, the confirmation message is a nicety but not a necessity.
Well, we should upgrade the kernel as ELF binaries are cool :-).
But if that's not possible, we'll just have to recompile irqtune to
create a.out binaries. This is,
perhaps, the only justification for rebuilding irqtune.
Just be sure
that /usr/src/linux/include is installed. The exact procedure for building
a.out binaries can vary with compiler revision, so it's important to check
the documentation on this (a parameter or two may need to be added).
make warp9
INSTALLATION:
install install prebuilt binaries in /sbin
custom create custom installation -- Normally _not_ required -- Try
install first (EQUIVALENT: kvers sbin install)
uninstall remove prebuilt binaries from /sbin
INSTALLATION OVERRIDES:
SBIN SBIN=/whatever specify installation directory (DEFAULT: /sbin)
INSTALL INSTALL=simple simple installation (DEFAULT: simple)
INSTALL=sh install as shell stub
INSTALLER installation program (bin/install) (DEFAULT: sbin/irqtune)
REBUILDING FROM SOURCE:
kvers create kernel_version
sbin rebuild binaries from source (full or partial)
FORCE FULL REBUILD (requires /usr/src/linux/include):
clean remove .o files to force full recompile)
warp9 rebuild all (EQUIVALENT: clean kvers sbin install)
Under the 2.0.X (and later) kernels, use of IRQ sharing will defeat IRQ priority because the serial port ISR's are installed as slow rather than fast interrupts (e.g. they don't use the SA_INTERRUPT flag).
#define IRQ_T(info) ((info->flags & ASYNC_SHARE_IRQ) ? SA_SHIRQ : SA_INTERRUPT)
and replace it with:
#define IRQ_T(info) (((info->flags & ASYNC_SHARE_IRQ) ? SA_SHIRQ : 0) | SA_INTERRUPT)
Under earlier kernels this is not a problem because the serial ISR was always installed with SA_INTERRUPT.
See the PPP man page and PPP-Howto for best information, but some recommended
options:
AT-like modems have a special character to escape from data mode to command mode. To avoid confusion here, we'll call this modem escape character a guard character.
The default guard character is '+' (decimal 43, hex 2B). Normally, 3 such characters are required within a special timing sequence.
Although it is unlikely, it is still possible that some PPP packets could
generate the guard sequence inadvertantly. To prevent this, we may want
to inhibit the generation of the guard character in a data sequence. To
do this,
we would add the additional PPP option:
Since '+' is a common ASCII character (PPP escaped characters generate two characters), we may wish to use a less common value for the guard character. For example, a less common value might be (decimal 200, hex C8). We would add an ATS2=200 command to our modem dialer script and change the PPP escape option to C8.
Some braindamaged PPP implementations do not handle PPP flag optimization!
The PPP protocol uses a flag byte to separate packets. Each packet
begins with a flag and ends with a second flag.
<FLAG> <PACKET (e.g. header, data, CRC)> <FLAG>
<FLAG><PACKET1><FLAG> <FLAG><PACKET2><FLAG> <FLAG><PACKET3><FLAG> ...
<FLAG> <PACKET1> <FLAG> <PACKET2> <FLAG> <PACKET3> <FLAG> ...
Although the PPP protocol requires implementations receiving packets to handle flag optimization, some broken PPP implementations do not understand it!
These implementations see the trailing flag, process the packet, then look for a fresh flag. They don't realize that the trailing flag of PACKET1 may perform double duty as the leading flag of PACKET2. They will ignore all data until they see a new flag (which, in this example, is the flag between PACKET2 and PACKET3). Thus, PACKET2 will be seen as noise data and be ignored. These implementations will only see only the odd number packets (e.g. PACKET1, PACKET3, PACKET5, etc.), resulting in a 50% packet loss!
Linux PPP implements flag optimization correctly and enables it by default. As charity to others, Linux does allow flag optimization to be turned off, but currently, this this requires the kernel to be rebuilt.
In Linux, to turn off flag optimization on transmit, do the following:
static int flag_time = OPTIMIZE_FLAG_TIME;
static int flag_time = 0;
Note: A better solution is to return the defective PPP implementation to the vendor and demand a refund or replacement!
By lowering the baud rate to something that is guaranteed not to drop data due to speed problems (e.g. 300 baud). If we get a consistent 50% loss at this low rate, this is almost certain proof of the flag optimization problem.
This is only necessary for the IDE driver. The SCSI driver has short disable windows by default. This will shorten the IDE interrupt disable windows.
Beware: Without this option, IDE disk activity will almost certainly cause serial data dropouts. If we have an IDE disk, this is mandatory.
This reduces the amount of bottom-half processing the system has to do at the expense of larger packets being sent. This may be helpful on slower CPU's or heavily loaded configurations.
Reducing the MRU/MTU to a minimum (296) reduces the bottom-half processing and flip-buffer latency at the expense of adding extra overhead bytes due to the reduced packet size. The optimal value will vary from configuration to configuration.
Beware: Start with 296 as the optimal may not be 1500.
The flip-buffer is a double buffer mechanism in the serial/tty drivers through which all data must pass. It has a fixed size of only 512 bytes. MRU/MTU greater than the flip-buffer size may create an internal race condition that may cause dropouts on slower CPU's or heavily loaded configurations.
Although irqtune will work surprisingly well with just about any kernel revision, the low level IRQ handlers and device drivers have been vastly improved in the 2.0.X kernels. This will only improve irqtune's effect.
The serial port baud rate should be high enough to support the maximum expected transfer rate--but no higher. Higher speed settings place extra strain on the CPU, increasing the likelihood of overruns.
For a 33.6 modem, the minimum baud rate would be 38400. However, with compression, the expected transfer rate can be as high as 6 KB/second. This would require a baud rate of 57600. This may strain the CPU, and since the transfer rate is nominally about 4 KB/second, a lower baud rate may be a good compromise.
The best way is to try several rates, then benchmark them to see which provides the best overall performance.
Note: Because of backward compatibility to older systems, we can't just set 57600 directly with stty, kermit, pppd, etc. Specify 38400 to these programs, and use the setserial program with an option of spd_hi. For ISDN speeds, use spd_vhi. Other options are possible so be sure to consult the manpage.
Well, first off, if PPP/SLIP was dying mysteriously, it will probably be more reliable.
Repeat this using irqtune and note the transfer times again.
NOTE: IRQTUNE just won't quit--if you want to test in the original mode again, reboot the system first.
It's a matter of probability. Performance measurement is as much art as science.