Short-Term Memory Loss

When I started preparing for the CCIE lab, I made a decision to build my own physical hardware lab at home.  Partly because I’m a geek and that’s the sort of thing we do, and partly because it made my life easier.  The ability to decide on the spur of the moment to fire up the rack and do an hour or two of labbing was very attractive to me, especially as most of the lab rental providers are US-based and work on 8 hour slots – this doesn’t always translate to friendly labbing hours here in the UK.

My lab is based loosely on the IPexpert topology – but built mostly using 1841s.  After a successful evening of labbing I’d shut down the rack and gone to bed.  The next day I’d fired it up again only to be caught by several of the 1841s getting stuck in a loop of stack traces and reboots.  The main error of any use was:

SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!

Houston, we have a problem…

My first step in diagnosing this was to look at the various routers that had booted successfully and then to look at the IPexpert “Initial” configs that I’d loaded on the previous evening.  The culprit was pretty obvious to spot within the first page of a “show run”:

memory-size iomem 15

The routers that had booted successfully had this set to 10% rather than 15%.

From cisco.com:
“To reallocate the percentage of DRAM to use for I/O memory and processor memory on Cisco 3600 series routers, use the memory-size iomem command in global configuration mode. To revert to the default memory allocation, use the no form of this command.”

Now it’s at this point that I should probably ‘fess up.  The vast majority of my 1841s are equipped with the default 128MB of DRAM and the 1841 IOS image that you need for CCIE studies typically requires 192MB.  It looks like this minor change in memory allocation was enough to make the difference between the device booting and not.

Of course at this point you have a router stuck in a loop.  I caught it at the next reboot and broke into rommon.  I had a quick look through the context-sensitive help and found a way to fix this:

System Bootstrap, Version 12.3(8r)T8, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2004 by cisco Systems, Inc.

telnet> send break
PLD version 0x10
GIO ASIC version 0x127
c1841 processor with 131072 Kbytes of main memory
Main memory is configured to 64 bit mode with parity disabled

Readonly ROMMON initialized
program load complet
monitor: command "boot" aborted due to user interrupt
rommon 1 >
rommon 2 > meminfo

-------------------------------------------------
Current Memory configuration is:
Onboard SDRAM: Size = 128 MB : Start Addr = 0x00000000
-----Bank 0 128 MB
-----Bank 1   0 MB
Dimm 0: Not Installed
-------------------------------------------------
Main memory size: 128 MB in 64 bit mode.
Available main memory starts at 0xa0015000, size 130988KB
IO (packet) memory size: 15 percent of main memory.
IO memory is configured by user
NVRAM size: 191KB
rommon 3 > iomemset
usage: iomemset [smartinit|5|10|15|20|25|30|40|50]
rommon 4 > iomemset 10

Invoking this command will change the io memory percent
WARNING:IOS may not keep this value
Do you wish to continue? y/n:  [n]:  y
rommon 5 > reset
<system boots normally>

Result!  Obviously once you’ve done this don’t forget to change the memory-size iomem in the config and wr mem, otherwise you’ll get bitten again tomorrow!