W3Perl speed

One w3perl weakness come from its speed to deliver stats. Of course, the package is not able to compute as fast as package written in C but the following data will show you that w3perl can do its job in a reasonable amount of time ... and as CPU speed are going faster and faster, time is less a critical value.

PackageLangageSpeed
(lines/s)
Time to process an
1 Gb logfile
Time to process an
1 Tb logfile
AnalogC23 0003 min 401 h
WebalizerC10 0008 min 202 h 20
AWstatsPerl4 10020 min 205 h 40
W3PerlPerl1 60052 min 14 h 30

An average 10 000 hits/day will produce a 1.2 Gb logfile/year and so stats will need around 1 h to complete.
If W3Perl is slower than AWstats, this is because W3Perl produce more detailed stats.
Don't forget that this is the time need to compute stats from scratch. Daily run are incremental and take a few minutes to complete whatever your logfile size.


Lines / sec
w3perl size

Stats have been computed on a Pentium III 450 Mhz 128 Mb under Linux.
The number of lines which have been rejected by the user configuration are not counted, it means in fact that only lines processed are displayed.

One point is one day, so a day took between 3 and 20 seconds to compute. To give an idea of access, there are around 10,000 hits a day or logfiles are around 1800 Kb large. Eight months of logfiles have been computed.


CPU timing If you want to send me your timing, feel free to email me your configuration.
A file is produced in your /data/cputime, you could send it to me.

Website Intervalle Logfile size
(compressed)
Hits / day CPU
(reverse dns off)
OS Speed
(lines / sec)
W3Perl 73 days 3,5 Mb 5,000 3 min 55 PIII 450 (Linux) 1000
W3Perl 421 days
(14 months)
11 Mb 2,000 16 min 20 PIII 450 (Linux) 1000
INRP 30 days 3,6 Mb 10,000 4 min 59 PIII 450 (Linux) 1000
INRP 244 days
(8 months)
47 Mb 10,000 36 min 00 PIII 450 (Linux) 1000

To give an idea, compressed logfiles are 90% smaller than uncompressed ones.


Hints If you want to increase speed :
  • Don't use the reverse dns option, querying dns server could take several seconds for each IP address. Results are stored in a file for the next calls but it really make things very slow. You could win a three to five speed increase.
    If your logfile have only IP address, the 'countries' stats could not be computed without dns reverse on.
  • Don't select too much pages in the @selection variable.
  • Use only one language output.
back