Tinypath.el --- Manage Emacs startup dynamically
- Preface Feb 1999 - How it all begun
- Overview of features
- How to set up your load path
- XEmacs and Emacs specific directories
- Finding load-path directories
- Gnus and other 3rd party packages
- Updating new lisp packages
- Duplicate files in path
- Symlinked directories are ignored
- Using cache
- Cache file and different Emacs versions
- Info file support
- Cygwin support (Win32 and woman.el)
- Faster Emacs configuration (Perl emacs-util.pl)
- Updating running Emacs
- Compressed lisp file support
- Trouble shooting
- Code note: General
- Code note: Where is that emacs package
- Code note: Internal optimizations
- Code note: *Messages*
- Code note: Custom
- Code note: Insinuating packages
- Code note: Elp profiling results
- Todo
Preface Feb 1999 - How it all begun
When you have set up your Emacs installation to your liking, a day comes when you decide that it's time to seriously reconsider the directory structure of your installed lisp packages. At start, it is customary to use simple file hierarchy where all private packages are installed under:
~/elisp ;; some people also prefer ~/lispComplete kits are usually installed directly under the root:
~/elisp/packages/bbdb-2.00.06/ ~/elisp/packages/psgml-1.0.3/ ~/elisp/packages/pcl-cvs-2.9.2/A more sophisticated way is to use symlinks to the latest versions, so that you don't have to change load-path every time you install a new version. It is only matter of updating the symlink:
~/elisp/packages/pcl-cvs/ --> ~/elisp/packages/pcl-cvs-2.9.2/ | This path is in the `load-path'In networked, where Windows ic oupled with Unix workstations via SAMBA, you may have mapped the H: disk to you Unix $HOME:
H: --> Unix $HOME \\SERVER\DIRECTORY\YOUR-LOGIN-DIRNow, there is a catch when you use Unix symlinks in $HOME/elisp and try to access the directories from Windows. Having set PC's HOME environment variable to point to H:, Emacs can start reading Unix $HOME/.emacs startup file, but you soon see simple error messages like "Can't load library xxx", which soon follow by bigger concerns: "autoloading xxx failed". The problem is the mounted H: disk. You see, PC's network mount can't distinguish symlinked directories from real directories, so all symlinked Unix directories in load-path are dead. And that's why most of the files cannot be found any more.
The conclusions
For cross platform solution it is best not to rely on symlinks, because they don't work over a Windows mount. Secondly, updating load-path should not be needed by hand after a new package installation, after a directory name change, after directory structure change, etc. A dream package would solve this all and do the hard work: "There, that is the root(s) of all Emacs lisp, go and sniff all the directories and update load-path"
That was what this package originally was all about. Nowadays it does a little more than that. Your load-path is updated automatically without any manual work. You only have to give the start ROOT path(s) of your installed lisp hierarchies in the file system. This package is highly effective: scanning thousands of files in a matter of seconds and once the cache has been created, it takes only a snap to load it in next sessions. All require and load commands execute also faster than previously, because the information about existing files is immediately available. The speedup is helped through advised functions.
Overview of features
Automatic load-path configuration
- Define list of root directories of your Emacs lisp and this package will recursively add directories which contain .el or .elc files to load-path
- A cache is utilized to remember previous scan and expired periodically. Using cache speeds up loading files considerably if you have many directories. The number of lisp directories doesn't affect the load performance any more. This is accomplished by using extra advice code in functions: load, load-library, require, locate-library and autoload.
- When Emacs becomes idle (some 15 minutes of idle time) the cache and load-path is validated for erroneous entries and rebuilt as needed. This feature should auto-detect changes in directory structure automatically and help semi auto-installing new directories (packages) for you.
- The load-path is optimized so, that users' files automatically take precedence first (~/elisp), next any other files found, and last the core Emacs files in the distribution. For example in previous Emacs versions, shipped speedbar.el used to be too outdated if used with JDE package. It had to be manually updated from the development site. That us the reason why core Emacs paths are at the then of load-path.
Automatic Info-default-directory-list configuration
- If you download packages that include Emacs info files, the Info-default-directory-list is updated at the same time as the load-path, when root directories are examined.
- No more manual updating of info files. The missing dir entry is created or updated as needed.
- You can update all new info files in your system by calling M-x tinypath-info-scan-Info-default-directory-list
If you have added new info files by hand, just call function tinypath-info-handler to update your Emacs and update the dir entry. After that reset infor with M-x tinypath-info-initialize
Win32 automatic manpath configuration
- In Unix systems the MANPATH enavironment variable contains directories where to find manual pages, but in Win32, there is no default MANPATH and M-x man does not work.
- If package woman.el (Included in latest Emacs versions) is along load-path, it is automatically configured to support to read manual pages. It replaces the M-x man command.
Win32 Cygwin environment support
- If cygwin1.dll (<http://www.cygwin.com/>) is in your path, automatic detection tries to find the Cygwin root and scan manual and info pages for use with woman.el Note: This feature s for native Win32 Emacs. Nowadays, there is also native Cygwin Emacs, which behaves just like the big brother *nix Emacs.
Compressed lisp file support
- Overloads commands load, load-library, load-file, require and autoload to accept jka-compr compressed lisp .el files.
- Primarily meant to be used in low quota accounts.
- Compress or decompress lisp files. You don't have to change a thing in your Emacs startup file, all will work as usual.
- Correctly handles aliased commands that turn out to be in autoload state.
How to set up your load path
The tinypath-:load-hook contains function tinypath-setup which starts examining all directories under load-path and tinypath-:load-path-root which is set to reasonable defaults of site wide and personal installations. If you keep all your lisp files under $HOME/elisp, then you do not need to configure anything for this package to work. Your load-path has been updated after this statement at the beginning of your $HOME/.emacs
(load "~/elisp/tiny/tinypath") ;; Or anywhere you have it installedIf you have many separate Emacs lisp root directories, like one for site-lisp and one for site-packages and one for your personal lisp files, then you have add those to variable tinypath-:load-path-root. Below there is an example for PC users, where the E: partition replicates identical Unix tree structure. We suppose for the moment that Cygwin is installed there. The following actually works for Unix too, because non-existing directories will be ignored:
(setq tinypath-:load-path-root '("~/elisp" "E:/usr/local/share/emacs/site-lisp/common")) (load "~/elisp/tiny/tinypath")XEmacs and Emacs specific directories
In spite of great effort from developers to make packages compatible for both Emacs platforms, there is always some packages that only works with Emacs or XEmacs. It is assumed that the site admin has created directories like these to keep the site-lisp installation clean:
;; This might be also under /opt/share/site-lisp ;; Refer to file hierarchy standard at ;; http://www.pathname.com/fhs/ /usr/share/emacs/site-lisp/common/ .. XEmacs and Emacs /usr/share/emacs/site-lisp/emacs/ .. only for Emacs /usr/share/emacs/site-lisp/xemacs/ .. only for XEmacsTo take care of the Emacs specific load-path setting, use code similar to this snippet. If you load the setup multiple times, the pushnew ensures that the directories are not added multiple times.
(require 'cl) (dolist (path ("~/elisp" "/usr/share/emacs/site-lisp/common" ;; Select Emacs or XEmacs specific installations (if (boundp 'xemacs-logo) "/usr/share/site-lisp/xemacs" "/usr/share/site-lisp/emacs"))) (when (stringp path) (pushnew path tinypath-:load-path-root :test 'string=))) ;; PLEASE COPY VERBATIM. THERE ARE OPTIMIZATIONS ;; THAT ACTIVATE If YOU ADD THE PATH (pushnew "~/elisp/tiny/lisp" load-path :test 'stringp=) (load "tinypath.el")In fact this package will check current emacs version and make sure that only correct directories are included to the load-path. If you simply instructed to search the whole site-lisp root /usr/local/share/site-lisp, and current emacs binary is "emacs", then all directories that contain path portion /xemacs woudl have been automatically ignored.
Building part of site-lisp using Net
If we continue talking a bit more about site-lisp, there is utility mywebget.pl at <http://perl-webget.sourceforge.net/>. It includes a mywebget-emacs.conf which contains knowledge where the various lisp developers' home pages are and how to download all known lisp tools that does not come with Emacs. If you have lot of disk space and you're interested in getting more lisp tools to go with your Emacs, follow the instruction laid out in this project's sourceforge page.
If you are further interested in Emacs packages, see CVS version control program available for Unix at <http://www.cvshome.com/> and for Win32 cvs will ship with the <http://cygwin.com> installation. With CVS you can track development of many Emacs projects including Gnus, BBDB, Mailcrypt etc. Cvs is minimizing network traffic by transferring only changes. Here is one suggestion where you could put all your Emacs Lisp Version control downloads:
/usr/share/emacs/site-lisp/net/cvs-packagesNow, the overall structure of whole site-lisp looks something like this:
ROOT/ ( /usr/share/emacs or equivalent ) | +--site-lisp/ | +--emacs/ | | ...Eamcs only files | +--packages/ | | +--pcl-cvs-2.9.9/ | | +-... and so on | +--win32/ | +--gnuserv/ | +-... and so on +--net/ | +--users/ | +-LispDeveloperA | +-LispDeveloperB | +-... and so on | +--cvs-packages/ | +--liece/ | +--lookup/ | +--ILISP/ | +--jess-mode/ | +--devel/ | +--emacro/ | +--tnt/ | +--cc-mode/ | +--mailcrypt/ | +--bbdb/ | +--gnus/ | +-... and so on +--common/ | ...COMMON for both Emacs and XEmacs | ======================================= | ...Packages that you find posted to the | ...gnu.emacs.sources and whose author's | ...do not have a homepage +--xemacs/ | ...XEamcs only files +--cvs-packages/ +--xemacs-packages/XEmacs 21.2+ core packages
The latest XEmacs versions ship with only the very basic installation. Lisp packages are distributed in separate archive xemacs-packages (nick named SUMO due to its huge size). There is also mule-packages and site-packages archives. A built-in heuristics tries to guess the location of these by looking under and near your XEmacs installation:
.../XEmacs/XEmacs-NN.N/xemacs-packages .../XEmacs/xemacs-packagesIf the archives have been installed elsewhere, you have to tell the location by defining following variable prior loading TinyPath. You can't put these to tinypath-:load-path-root because this is special information that needs to present during the very initial boot-up to find crucial packages like jka-compr.el.
(setq tinypath-:core-emacs-load-path-list '("/usr/share/site-lisp/xemacs/xemacs-packages" "/usr/share/site-lisp/xemacs/mule-packages" "/usr/share/site-lisp/xemacs/site-packages"))Finding load-path directories
If you only used default $HOME/elisp directory for your files, the tinypath-:load-path-function starts recursively searching all the directories under the root(s) tinypath-:load-path-root. Not all directories are counted in when the search descends below the root(s). Variable tinypath-:load-path-ignore-regexp decides if the directory should be ignored. By default:
- Package's additional subdirectories like texinfo, tex, doc, etc, misc, RCS, CVS, .svn (Subversion), MT (monotone version control), zip are ignored.
- Any temporary directories named .../t/ .../T/ .../tmp* .../temp* are ignored.
- Directories that do not contain any files ending to .el or .elc are ignored.
Gnus and other 3rd party packages
Note: In latest version of this utility Gnus is treated specially. All Gnus versions are detected along load-path and the very latest Gnus version is installed to your load-path. This is based on the knowledge in the gnus-version variable and the heuristics will pick the newest for you. You actually do not have to do anything else, but to drop latest Gnus somewhere, to be able to use it immediately.Under the hood - What follows, is old documentation.
It is important to understand how this package works: It caches every possible lisp directory it can find. Now, if you have installed private copy of Gnus, say in ~/elisp/cvs-packages/gnus, there is a problem, because Emacs distribution also includes Gnus. There is NO WAY TO TELL OR CHANGE path order when the cache is in use. This is a design decision and cannot be changed. The old trick, where a new directory was added in front of load-path, will not work because everything goes through cache. What you need to do instead, is to tell that the "other" Gnus should be ignored during cache creation, so that it is completely unknown.
Solution: ignoring directories
There is very simple way. Put your regular expression to tinypath-:ignore-file-regexp-extra and it will tell which directories to ignore. Naturally you must put the lisp code before you load package.
(setq tinypath-:load-path-ignore-regexp-extra "\\|[/\\]x?emacs[/\\0-9.]+[/\\]lisp[/\\]gnus") ;; PLEASE COPY VERBATIM. THERE ARE OPTIMIZATIONS ;; THAT ACTIVATE If YOU ADD THE PATH (require 'cl) (pushnew "~/elisp/tiny/lisp" load-path :test 'stringp=) (load "tinypath.el")[For advanced Lisp programmers] You can add ignored gnus directory to tinypath-:load-path-ignore-regexp via tinypath-:load-path-ignore-regexp-hook. When the hook is run, the default value for tinypath-:load-path-ignore-regexp is already available. In hook, append regular expression that excludes the Gnus directory. Here is an example; make sure that you don't add the regexp multiple times. The multiple invocations is protected by setting a plist property and checking it. The ugly [\\/] makes the regexp compatible with both Unix and win32 paths. System directories in Unix are typically /emacs/NN.NN/ and in win32 /emacs-NN.NN/, that's why added "-".
(add-hook 'tinypath-:load-path-ignore-regexp-hook 'my-tinypath-:load-path-ignore-regexp-hook) (defun my-tinypath-:load-path-ignore-regexp-hook () ;; Do this only once (unless (get 'my-tinypath-:load-path-ignore-regexp-hook 'set) ;; mark as done. (put 'my-tinypath-:load-path-ignore-regexp-hook 'set t) (setq tinypath-:load-path-ignore-regexp (concat tinypath-:load-path-ignore-regexp "[/\\]x?emacs[/\\0-9.]+[/\\]lisp[/\\]gnus"))))#todo: What about XEmacs public/private Gnus installations?
Updating new lisp packages
Suppose you have installed a new version of a package:
~/elisp/gnus/pgnus-0.74/ ~/elisp/gnus/pgnus-0.95/ ;; NEWBoth these directories end up being added to the load-path, but that is not preferable. It is the latest version that should be in the load-path. The solution is to move the old versions under some name that will be ignored by default. It is recommended that a backup of previous packages are renamed to start with a word "tmp-". All directories that start with prefix tmp are ignored.
% mv ~/elisp/gnus/pgnus-0.74/ ~/elisp/gnus/tmp-pgnus-0.74/ ====However if you update package in a site-lisp directory, there may be a distant problem that somebody needs older version of the package. If you made the backup like above, that user cannot load the old package any more, because it doesn't show up in load-path
There is no easy answer to keep old packages. Admin could announce that: "new version has been installed in DIR, the old one is in TMP-OLD-DIR" and have users manually arrange their load-path if needed. Following lisp command would solve their setup. The statement below adds the old directory to the beginning of load-path and thus load commands would find the old version of the package first.
(load "~/elisp/tiny/tinypath") (pushnew "TMP-OLD-OLD-DIR" load-path :test 'string=) (tinypath-cache-regenerate)Remember to mention to users that they need to update cache with tinypath-cache-regenerate (called with prefix argument) to see the changes.
Duplicate files in path
If you have accustomed to putting your path to specific order, you have to rethink your strategy if you are going to use this package. The philosophy behind creating this utility was: YOU SHOULD NOT NEED TO DO MANUAL WORK TO UPDATE PATHS. This means that the order of the paths must not be significant. Now, you may face a situation where you install a library or package that contains a file, which has already been installed to your hierarchy. Take for example, smtpmail.el:
/usr/bin/emacs-20.4/lisp/mail/smtpmail.el /usr/share/site-lisp/common/packages/semi/flim-1.12.1/smtpmail.elWe have a problem here if FLIM's smtpmail.el is not compatible with the one in Emacs. If it is, then there is no problem. Either one can be loaded, and the load-path order does not matter. But you don't know that before you get error "function smtpmail-xxxx not defined" and you start investigating with (locate-library "smtpmail") which package is actually active.
Acer activating this package, please investigate your path with [C-u] M-x tinypath-cache-problem-report and see if you find duplicate entries. Investigate each one and possibly move the file to another name or remove older ones. E.g. in the above situation, the cure might be moving FLIM's smtpmail.el under name flim-smtpmail.el so that it doesn't get loaded with (require 'smtpmail). The best is to contact the maintainer(s) and tell them about conflicts. Here is a sample of one generated problem report:
imenu.el 323 34073 1998-05-07 16:28:08 /usr/share/site-lisp/common/other/ 910 37169 1999-12-04 02:47:58 /usr/share/site-lisp/common/programming/java/jde/jde-2.1.6beta13/lisp/ 1350 38663 1999-11-28 01:14:38 /usr/bin/emacs/gnu-emacs/emacs-20.4.1/lisp/ base64.el 515 9943 1999-12-11 19:15:20 /usr/share/site-lisp/common/packages/gnus-5.8.2/lisp/ 807 9892 1999-11-15 00:00:12 /usr/share/site-lisp/common/packages/w3-4.0pre.46/lisp/Explanation: User had used imenu as a separate package since early Emacs versions in "other" directory. Now the latest Emacs ships one, so it is best to delete the previous one other/imenu.el. Keep on eye on the numbers here: The lower, the more close it is to the beginning of cache when the directories were searched. The package with lowest score will get loaded. Another package, base64.el seems to be problematic too. But because Gnus path has lowest score, it will get loaded before w3's base64.el. This is good, because Gnus contains the latest version of base64.el. In the buffer tinypath-report-mode is turned on to manipulate reported lines. Unnecessary files can be deleted with Control-shift-mouse-1 or C-c C-d.
Symlinked directories are ignored
It has been the tradition to use symlinks a lot in Unix to arrange easy access to versioned packages. Like how to ~/elisp/gnus/ no matter what version is currently installed.
ln -s ~/elisp/packages/gnus-N.NN ~/elisp/packages/gnusThis package however skips those symlinks and records the absolute path name to the load-path. There are couple of points: a) it is more instructive to peek the load-path to actually see what versions have been installed to the Emacs b) The symlinks are error prone since there may be several symlinks that lead to same directory and c) symlinks don'e work in heterogenous environments where Win32 and Linux and Unix is networked together. To migrate to this package you need to examine your symlinks and remove them.
If you have drawn a symlink to the the current directory from SEPARATE directory, then that directory will never be seen:
ln -s ~/some-disk/elisp/artist-1.1/ ~/elisp/packages/artist-1.1To solve this, instead either a) move the package physically under the ~/elisp/ from the ~/some-disk/elisp/ so that the recursive search will record it or b) add the separate directory ~/some-disk/elisp to the variable tinypath-:load-path-root.
Using cache
Now when you're freed from update burden of the directories in your disk, you can concentrate organizing the files under sensible directories. Here is an example how the organizing could go:
~/elisp/users/kevinr/ Kevin Rodger's files ~/elisp/users/ilya/ Ilya Zakharevich's files .. ~/elisp/packages/bbdb-2.00.06/ Version-ed packages ~/elisp/packages/psgml-1.0.3/ ~/elisp/packages/pcl-cvs-2.9.2/ ~/elisp/packages/tiny-19990215/ ... ~/elisp/other/ All single add-on packagesAll these paths in load-path and you can imagine how slow a standard Emacs would become: it takes even more time to find some package xxx, when Emacs sees a call (require 'xxx), because Emacs must start looking into every single directory under load-path until it can determine if it can or cannot load the asked package. This utility will store all lisp files in cache, and it is activated by default. The variable tinypath-:cache-expiry-days controls the interval when it is concluded that a new tree recursion is needed. If you install new packages during those non-expiry days, it is best to call C-u M-x tinypath-cache-regenerate to build up to date image of your files and load-path directories.
If you want one short advice: always call tinypath-cache-regenerate after any lisp file or directory update.
Cache file and different Emacs versions
It is important that each Emacs loads correct cache file. The cache file's name is derived from the emacs version and emacs type, which can be "xemacs", "win32-xemacs", "emacs" or "win32-emacs".
tinypath-:cache-file-prefix + EMACS-TYPE + HOST + EMACS-VERSION + tinypath-:cache-file-postfix ~/elisp/config/emacs-config-tinypath-cache-win32-HOST-emacs-20.4.1.el.gz ========================================== ====== prefix postfixUnix hosts and NFS mounts
In Unix environment, it is also common that several hosts are NFS mounted so that your home disk is available from every server. The available programs are not usually NFS mounted, but programs are stored locally on each server's own disk. Now, there would be a problem if you logged to host A and started tinypath.el which had made cache in on host B, because A does not have the same directories as B did (site-lisp). This has been taken care of by including hostname part in the cache file name. For each host, a separate cache file is created. Now, suppose all the Unix hosts are same brand, say Sun OS, Linux, or HP-UX and a good administrator has separated the programs and the data in their own directory structures. Furthermore, these structures are NFS mounted and thus visible to the remote machines. In this scenario, it would not really matter to which host you log into, because you would always see the same programs and site-lisp directories and there would not be need for host specific cache files. In that case, disable the HOST word by setting this variable to nil:
(setq tinypath-:cache-file-hostname-function nil)Info file support
In addition to updating the load-path, the recursive function has a chance to search for installed info files as well. When you keep all your site lisp under one directory, it is not uncommon that the bigger packages include documentation files in info format as well. Like:
/usr/share/site-lisp/emacs/pcl-cvs-2.9.9/ /usr/share/site-lisp/common/packages/psgml-1.2.1/One possibility is that after you download and uncompress a package, you copy the info file to some central directory where you keep all you info files. This is lot of manual work. (Never mind that in Unix you might use Makefile to install everything, in Win32 it's all manual work). This package does the same job by looking for directories that either have info files or a central info repository called dir. If the dir file has all the info files up to date, nothing is done. In other cases:
- If the central dir in the directory does not exits, it is created.
- If dir does not contain entry for info file, it is added. The entry name is derived from the filename.
The Info-default-directory-list is updated to include any new directory locations and they are saved to same cache file. When you call C-h i you will see the new info entries. Easy and maintenance friendly. No need to worry about supplied info files any more, they are automatically integrated to your Emacs. If you have installed any new packages to your system, Emacs packages or Unix packages that installed something with "install -c", it is best to update your info files with M-x tinypath-info-scan-Info-default-directory-list. This is also called if you call: C-u M-x tinypath-cache-regenerate
Cygwin support (Win32 and woman.el)
It is common that Emacs in Win32 environment is coupled with <http://www.cygwin.com> toolkit which contains all the manual pages for the unix commands and possibly new info pages. This package will locate cygwin1.dll file along PATH and recurse whole cygwin installation root to find new entries that can be used inside Emacs. In theory this all should happen automatically and the only thing you have to do is to ensure that you have proper PATH settings at your OS level before this package is started. If Cygwin /bin directory in in PATH, tinypath-:extra-path-root will get set to a correct value at boot time.If you have more places where you keep Unix tools which contain more manual or info pages, like Reed Kotler (old Unix-like env) http://www.reedkotler.com/ you must manually set variable tinypath-:extra-path-root to the list of search root directories. If you set this yourself, you must also include the cygwin installation root directory
(setq tinypath-:extra-path-root '("e:/unix-root/cygwin" "e:/unix-root/reed-kotler" ...))Package woman.el will be configured automatically if it is along load-path to handle manual page viewing with command M-x man. Please make sure that you do not destroy the pre-defined woman-manpath in your Emacs startup files with lisp commands or the efforts to find out new manual pages are thrown off the window. Search you startup files for anything that looks like setq, defvar, defconst: (setq woman-manpath ... and change the code to add to the variable instead:
(require 'cl) (dolist (path '("one" "two" "three")) (pushnew (expand-file-name path) woman-manpath :test 'string))Faster Emacs configuration (Perl emacs-util.pl)
Indication of this feature at startup is following message, where EXT refers to externally launched process which must be waited until further processing is done.
TinyPath: EXT Process running ... [may take a while]As this package evolved and more support was added to various environments, like Cygwin, which requires traversing hundred of directories to find out if they contained info or manual pages, it came evident that Emacs Lisp method was too slow. An alternative method was developed using Perl language and written in emacs-util.pl which can traverse directory hierarchies to find relevant directories for the setup. This interface is automatically used if two conditions are met in current environment:
- Binary perl must be along PATH. (according executable-find)
- perl script emacs-util.pl must be along PATH. Either copy the file to suitable place or include Tiny Tool's /bin directory to your PATH (This is not the whole story, see "Internal optimizations").
If all goes well, a call-process to the utility script will return the file hierarchies much faster than the Emacs Lisp ever could. The difference is that you don't see the traversing progress as you would if Emacs Lisp did the same thing. The command line arguments passed to the utility scripts can be found from the Message buffer and you can run the program yourself if you think that it returns incorrect listing. Print the script help with following command:
% perl emacs-util.pl --helpHere are some performance statistics of the perl script in action. (Use --verbose argument to see the statistics)
- Intel 400MHz, IBM GXP 80G IDE disk, whole Cygwin installation scan: 3 min 46 sec, dirs: 2373, files: 35 271
- Same PC, but this time site-lisp directory, subset of Cygwin hierarchy at /usr/share/site-lisp took: 0 min 13 sec, dirs: 648, files: 8750
Let's consider one scenario that you may encounter if you intend to use Cygwin similarly as the big brother Linux. Let's suppose that you have dedicated a disk portion where you intend to duplicate whole Linux-like directory hierarchy. You have ROOT under which you keep all the files, including anything that is Cygwin-related.
E:/usr/share/site-lisp Emacs lisp as outlined earlier E:/usr/share/site-perl Perl packages and scripts E:/usr/share/site-php PHP code E:/usr/share/site-cvs Various other external CVS C-packagesThe default heuristics ti::win32-cygwin-p should find cygwin1.dll installed and report that Cygwin root is E:/ This means that tinypath-:extra-path-root will get set for you when package loads. Suppose further that you have set variable tinypath-:load-path-root to point out suitable locations in E:/usr/share/site-lisp. It would seem that this combination means that the hierarchies would be traversed multiple times, since the Cygwin root already includes all the rest:
E:/ Cygwin root E:/usr/share/site-lisp/emacs For this emacs... E:/usr/share/site-lisp/common Emacs and XEmacs compatible treeDon't worry. The Perl utility is smart enough to reduce this to search only E:/ and discard other roots as redundant. Hm, what if other lisp files are found outside of the E:/usr/share/site-lisp/, because it searches every dir starting from E:/ Say:
E:/tmp/try/some-file.elWill the directory E:/tmp/try/ reported as lisp load-path candidate and added to search list? Yes and no. Yes, it will be reported, but no, it will not be added to the load-path because it doesn't match the initial user's idea where to look for lisp files. If you pump up the tinypath-:verbose to level 5, you can see PATH-NOK messages labeled "candidate" to indicate those rejections. Only files that reside under tinypath-:load-path-root directories are counted in.
Updating running Emacs
Suppose you have downloaded the latest versions of packages X, Y and Z and you want your current emacs's paths updated, call this function:
M-x tinypath-cache-regenerateTake a bit of skepticism: It is a fortunate event if it all worked that easily. You see, you already have several packages loaded in your Emacs and they are using the "old" code. Now you wiped the old directories away and told Emacs to look for only "new" directories. After a while you may run into bizarre dependency problems. I recommend that after any major package update, which contains several of files (like Gnus), you:
- Install package and regenerate cache in current Emacs session with M-x tinypach-cache-regenerate.
- Save your current Emacs buffers (see desktop.el, tinydesk.el)
- Quit, restart Emacs and restore your working desktop.
Compressed lisp file support
In order to use the full compression support for autoload functions as well, set variable tinypath-:compression-support to symbol all. the default support only handles require and load commands. the variable must be set before rhis utility package is loaded.Jka-compr and this package
jka-compr has native support to un/compress any file that have specific extensions. The handling is done via file-name-handler-alist and commands like these will load properly including any autoloads.
(load "my-el.gz")The obvious problem is that you have to manually go and change all you load commands so that they end in .gz so that jka-compr takes care of loading. What if you later uncompress the file? You have to go and update all the load commands in you Emacs startup files. This isn't very nice, since you should be able to un/compress elisp files whenever you wish and still have permanent statement like one below. Basically this is what the compression support here is all about; you don't have to worry if the file is compressed or not when these advised functions are in effect. The following statement will always work:
(load "my-el")How the compressed loading works
- When user request load FILE, try to find some compressed file that JKA knows about by adding extensions ".gz" and ".Z" and whatever user has configured JKA to handle. LIMITATION: only .gz .bz2 and the like that compress one file at a time is currently supported. Don't try using .zip or similar.
- If the FILE is absolute path, then look from that directory only.
- If no directory is given, find the file along the path.
- If there was somewhere a compressed file, just load it (because JKA will transparently uncompress it), eval it, and kill the buffer.
- If NO COMPRESSED file was found, just follow normal emacs rules.
Note: Why you should not prefer compressed .elc files
The purpose of compression support is to make it possible to have more useful lisp files in an account that has a limited disk space (quota). Many Unicersity student accounts have this limitation. Keeping lisp files in compressed format saves quite a much disk space.
- Plain text, lisp .el, files may compress better.
- The documentation in comments is important, e.g all the instruction to use the file are there. Byte compiling strips away documentation.
- In order to debug or send bug reports you have to use .el files, the .elc files are useless.
- The performance ratio that the .elc files offer may not be a crucial factor (many times you couldn't tell).
Note: advised emacs commands
The adviced functions can be further adviced, but if the redefined function uses interactive-p test, it will not indicate user call (like M-x load-library). The reason why the advised functions detect it, is that advice.el's ad-do-it macro cannot pass the interactive flag information to the original functions.
Trouble shooting
There is no denying it, this package is dangerous. When something goes wrong, it really goes wrong and your Emacs may be messed up completely. So, here are some trouble shooting tips, that you might want to try to rescue the situation or understand what is going on. The most usual blame is the cache content which does not contain the correct or up to date information.Package is not found or loaded?
Please confirm that the file location is known and is in right directory by calling M-x locate-library. If the result is not correct, please check tinypath-:load-path-root and tinypath-:extra-path-root. Try to remedy the situation, regenerate cache with M-x tinypath-cache-regenerate.
You don't know what particular package is causing troubles
Go to the Message buffer and clear it (C-x h followed by C-w). Run the path generation engine with debug M-x tinypath-debug-external-helper and study the output. It may be ignoring some files that you think should be included. Please check content of tinypath-:load-path-ignore-regexp and tinypath-:load-path-ignore-regexp-extra.
You need to see the internals
Call function tinypath-cache-file-find-file to display the current cache and use C-s and C-r to search entries in the file. Remember that you must not modify this file, because any changes you do, will get overwritten next time the cache is regenerated. The problem is somewhere else if you can see incorrect settings in the cache file.
Code note: General
Because this package is among the first that is loaded from Emacs startup file, It contains copies of some functions from TinyLib; libraries, to make the package independent until the point where the load-path has been set up and other libraries are available. In the code you may find marks "#copy:" which indicate code that has been copied/simplified to be used here. Autoloads statements in this package defer loading functions until the end is reached and load-path is determined and the rest of the functions can be loaded from the libraries.Code note: Where is that emacs package
If you ever need to know the location of a package that Emacs would load or has loaded, while this utility is in effect, use this call:
(insert (tinypath-cache-p "gnus.el"))In fact the regular call yields same result, because locate-library is adviced:
(insert (locate-library "gnus.el"))More easily, with tinylisp.el, which takes advantage of tinypath.el cache, you can load any emacs package for editing with command:
M-x load-library RET tinylisp RET M-x tinylisp-library-find-file (tinypath cache)Lisp Library: gnus.el RETAlternatively there is mode hot-keys $ l f and $ l p :
M-x load-library RET tinylisp RET M-x tinylisp-install M-x tinylisp-mode (in *scratch* buffer, see "E" in modeline) $ l f (tinypath cache)Lisp Library: gnus.el RETCode note: Internal optimizations
In the installation section it is instructed that the location of the package is pushed into the load-path before the package is loaded:
(require 'cl) (pushnew "~/elisp/tiny/lisp" load-path :test 'stringp=) (load "tinypath.el")Please follow this instruction. The reason is that program tries to use most efficient code to boot everything up and the first thing it can do is to check the location where it has been saved. This package will use this information to assume that the Perl program is available at ~/some/path/bin/emacs-util.pl. If that fails, the Perl program is searched along exec-path. This is usually desirable, situation because every new installation includes newer version of Perl program and the one at exec-path may not be up to date. The perl code will speed up booting compared to pure Emacs Lisp implementation. In addition the Perl code section in this file (often referred as "external") has extra features included.
Code note: *Messages*
This package will print loads of messages to Emacs "Message" or XEmacs " Message-Log" buffer. This is a design decisions so that execution can be easily traced during Emacs load time. It also help reporting errors. The default tinypath-:verbose 3 will log the most important messages. Even if you set the level to 0 or nil, still some messages are displayed. Have a look at Message buffer if you have not much used it before. You may find interesting information to debug some of your own mis-configurations, like stale directories in exec-path.Code note: Custom
If you have Emacs that does not contain custom.elc (Yes, it must be in compiled format, be sure to check), you can download Noah Friedman's excellent custom emulation package cust-stub.el at http://www.splode.com/~friedman/software/emacs-lisp/ You have to load it from absolute location before loading this packages like this:
(load "~/elisp/noah/cust-stub") (load "tinypath")Code note: Insinuating packages
Some packages can be auto-configured when the perl script reads the contents of the directories. Like package woman.el which needs to know the location of man path directories. For other packages there are different "installations". Gnus is one interesting example: Every Emacs and XEmacs release comes with Gnus version, which is usually outdated and many install Gnus privately. The problem is multiple Gnus versions in the load paths and the wished situation is that there would be only the latest. there is experimental code to find out which of the Gnus packages along load-path is the latest and after making hopefully right decision (according to gnus-version-* variable) the other Gnus locations are hidden by modifying load-path and tinypath-:load-path-ignore-regexp. This is a complimentary method to that suggested in the manual section "3rd party packages".Code note: Elp profiling results
The profiling results were run using method below. It must be note, that the tinypath-external-* is the time when the external perl program examines all the directories, so EXT time is not significant because it varies from system to system. The tinypath-external-setup-parse-data is the actual time spent in parsing the returned data. The functions that are called most of the time are the ones that must be kept on eye on and they seem to perform very well. Immediate below are the most important functions that perform the Parsing after the perl has returned results (these are not from the total listing, but after tweaking). The listing below represents timing results somewhere around 2001:
tinypath-external-output-parse 1 4.89 4.89 tinypath-external-output-parse-1 5 1.09 0.21 tinypath-external-output-parse-1-cache 1 3.79 3.79
tinypath-external-setup-parse-data 1 5.77 5.77 tinypath-external-setup-1-load-path 249 0.70 0.002 tinypath-external-setup-1-man-path 44 0.0 0.0 tinypath-exec-path-append 73 0.92 0.012 tinypath-info-handler 31 8.46 0.27 tinypath-external-setup-cache 1 0.0 0.0These timing results was taken 2003-05-18 running Cygwin XEmacs 21.4.10, Pentium 400 Mhz. These profiling results are from the initial boot phase, before cache is loaded. It's pretty fast.
(setq tinypath-:install-flag nil) (load "elp" (load "tinypath") (elp-instrument-package "tinypath-") ;; Now run the boot phase ONLY (tinypath-load-path-initial-value tinypath-:core-emacs-load-path-list) Function Name Count Elap Ave =============================================== ===== ===== === tinypath-load-path-initial-value 1 0.477 0.47 tinypath-load-path-add-subdirs 1 0.463 0.46 tinypath-directory-subdirs 1 0.451 0.45 tinypath-emacs-root-directory 1 0.008 0.00 tinypath-emacs-root-by-load-path 1 0.008 0.00 tinypath-emacs-core-path-p 119 0.004 3.36 tinypath-expand-file-name 5 0.001 0.00 tinypath-load-path-initial-value-xemacs 1 0.001 0.00 tinypath-load-path-string-match 1 0.001 0.00 tinypath-win32-p 5 0.0 0.0 tinypath-emacs-versions 1 0.0 0.0Theses timing results was taken 2003-05-18 running Cygwin XEmacs 21.4.10, Pentium 400 Mhz. The cache with 4500 directories was loaded from configuration file. In this case tinypath-:cache-file-postfix value was '.el'. The timing information was tested and generated with:
- C-x C-f tinypath.el RET – toad read tinypath.el to Emacs
- M-x load-library RET tinylisp.el RET
- M-x turn-on-tinylisp-mode RET
- $ - to eval current buffer
- $ e I to instrument everything (Wtih empty value, scan buffer)
- M-x tinypath-cache-regenerate RET
- $ e s to show results
Function Name Count Elap Ave =============================================== ===== ===== === tinypath-install 1 6.812 6.81 tinypath-cache-setup 1 5.76 5.76 tinypath-setup 1 5.76 5.76 tinypath-directory-list-clean 7 3.756 0.53 tinypath-cache-file-load 1 2.552 2.55 tinypath-load-path-clean 1 2.272 2.27 tinypath-cache-file-need-sync-p 1 1.932 1.93 tinypath-load-path-not-in-synch-p 1 1.932 1.93 tinypath-exec-path-clean 2 0.679 0.34 tinypath-exec-path-check-verbose 2 0.597 0.298 tinypath-exec-path-check 2 0.594 0.297 tinypath-load-path-merge 1 0.364 0.364 tinypath-Info-default-directory-list-clean 1 0.218 0.218 tinypath-file-remove-trailing-slash 825 0.121 0.000 tinypath-cache-mode 1 0.082 0.082 turn-on-tinypath-cache-mode 1 0.082 0.082 turn-on-tinypath-cache-mode-maybe 1 0.082 0.082 tinypath-ti::advice-control 1 0.081 0.081 tinypath-install-timer 1 0.007 0.007 tinypath-cache-file-name 2 0.006 0.003 tinypath-exec-path-from-path 2 0.006 0.003 tinypath-ti::compat-timer-cancel-function 1 0.005 0.005 tinypath-ti::compat-timer-elt 2 0.004 0.002 tinypath-cache-warn-if-not-exist 1 0.004 0.004 tinypath-cache-file-old-p 1 0.004 0.004 tinypath-days-old 1 0.002 0.002 tinypath-cache-status-string 2 0.002 0.001 tinypath-cache-status-message 1 0.002 0.002 tinypath-advice-instantiate 1 0.002 0.002 tinypath-expand-file-name 3 0.001 0.000 tinypath-ti::compat-timer-cancel 1 0.001 0.001 tinypath-use-compression-maybe 2 0.001 0.0005 tinypath-exec-path-append 1 0.001 0.001 tinypath-win32-p 6 0.0 0.0 tinypath-emacs-versions 2 0.0 0.0 tinypath-ti::date-time-difference 1 0.0 0.0 tinypath-eval-after-load 1 0.0 0.0 tinypath-time-string 2 0.0 0.0 tinypath-file-compressed-p 2 0.0 0.0 tinypath-emacs-lisp-file-list-cache-clear 1 0.0 0.0 tinypath-autoload-file-name 1 0.0 0.0 tinypath-autoload-require 1 0.0 0.0 tinypath-cache-p 1 0.0 0.0 tinypath-cache-file-hostname 2 0.0 0.0 tinypath-load-path-root-changed-p 1 0.0 0.0Same timing test as above, but now using compiled cache file at Emacs startup. In this case tinypath-:cache-file-postfix value was '.elc'. The speedup is 50%, rducing the load time to mere 3-4 seconds. Notice the dramatic change in tinypath-cache-file-load: 0.5 seconds vs. 2.5 seconds non-compiled.
Function Name Count Elap Ave =============================================== ===== ===== === tinypath-install 1 3.305 3.30 tinypath-cache-setup 1 2.017 2.01 tinypath-setup 1 2.017 2.01 tinypath-directory-list-clean 7 1.608 0.22 tinypath-load-path-clean 1 0.904 0.90 tinypath-advice-instantiate 1 0.784 0.78 tinypath-cache-file-load 1 0.549 0.54 tinypath-exec-path-check 2 0.506 0.25 tinypath-exec-path-check-verbose 2 0.506 0.25 tinypath-load-path-not-in-synch-p 1 0.368 0.36 tinypath-cache-file-need-sync-p 1 0.368 0.36 tinypath-exec-path-clean 2 0.326 0.16 tinypath-exec-path-from-path 2 0.154 0.07Todo
- In theory it is possible to load remote files with ange-ftp/EFS in manner of load-library RET /user@host:/path/to/file but this has never been tested.
- It theory it is possible to add /user@host:/path/to/dir/ to load-path, but that has never been tested.
This file has been automatically generated from plain text file
with Perl script t2html.pl v2004.0330
Last updated: 2004-04-05 13:37