Understanding Buffers

             While working with multiple customers have observed bandwidth utilization greater than 70-75% percent results in reduced performance (Packet loss, latency etc). Increasing the bandwidth is a solution, however might not be necessary or immediately practical. One way to resolve marginal over utilization problems is to control how the router uses data buffers. Extra caution should be taken while dealing with buffers as this might affect the device hardware and overall performance if configured incorrectly. 

Other way around buffer tuning is required if we are seeing buffer misses/drops on the serial interface.

The IOS buffers are used for two major reasons.To handle traffic terminating at the router. and When packets are being process switched. 

If there are no buffers available, and fast switching is enabled, there is a buffer failure and the packet is dropped. When the buffer pool manager process detects a buffer failure, it "creates" a new buffer to avoid future failures.

One of the following can be used to control how buffers are used:

• Adjust parameters associated with system buffers.

• Specify the number of packets held in input or output queues (hold queues).

• Prioritize how traffic is queued for transmission (priority output queuing).

1. Tuning System Buffers

Two types are buffers are available in Cisco devices.

*System buffers which are directly configurable. Associated with the main system memory and are allocated to different-size memory blocks.

*Hardware buffers are used as receive and transmit buffers associated with each interface and are dynamically managed by the system software itself.

Show buffer is the command that helps you find out the status of all the buffers in the device.

Router# show buffers

Buffer elements:

399 in free list (500 max allowed)

87777491 hits, 0 misses, 0 created

Small buffers, 104 bytes (total 120, permanent 120):

114 in free list (20 min, 250 max allowed)

70005538 hits, 6 misses, 2 trims, 2 created

Middle buffers, 600 bytes (total 90, permanent 90):

88 in free list (10 min, 200 max allowed)

25696696 hits, 27 misses, 27 trims, 27 created

Big buffers, 1524 bytes (total 90, permanent 90):

90 in free list (5 min, 300 max allowed)

8214549 hits, 14 misses, 366 trims, 366 created

Large buffers, 5024 bytes (total 5, permanent 5):

5 in free list (0 min, 30 max allowed)

15018 hits, 12 misses, 16353 trims, 16353 created

Huge buffers, 18024 bytes (total 3, permanent 0):

2 in free list (0 min, 4 max allowed)

297582 hits, 17 misses, 30 trims, 33 created

0 failures (0 no memory)

Six buffer pools are available:

Small—104 bytes buffers
Middle—600 byte buffer
Big—1524 byte buffers
VeryBig—4520 byte buffers
Large—5024 byte buffers
Huge—18024 byte buffers

 15-25

*Total = total number of buffers in the pool (used + unused buffers).

*Permanent = Identifies the permanent number of allocated buffers in the pool. These buffers are always in the pool.

*Free list = The number of buffers in free list is the  available buffers. When a buffer request comes in, a buffer from the free list is allocated.

*Min = Minimum number of buffers that the route processor  should attempt to keep in the free list. If the number of buffers in the free list falls below the min value, the RP attempts to    create more buffers for that pool.

*Max = Maximum number of buffers allowed in the free list. The max allowed parameter prevents a pool from extra buffers that it doesn’t need anymore, and frees this memory back to the system for further use. If the number of buffers in the free list is greater than the max allowed value, processor should attempt to trim buffers from the pool.

*Hits = Number of buffers that have been requested from the pool. Helps identify which pool has maximum requests.

*Misses = Number of times that a buffer has been requested and that the processor detected that additional buffers were required. Processor tries to create additional buffers.

*Trims = Number of buffers that the RP has trimmed from the pool when the number of buffers in the free list exceeded the number of max allowed buffers.

*Created = Number of buffers that has been created in the pool. The processor creates buffers when demand for buffers has increased until the number of buffers in the free list is less than min buffers or a miss occurs because of zero buffers in the free list.

*Failures= Number of failures to grant a buffer to a requester even after attempting to create an additional buffer. Equal to number of packets dropped due to lack of buffer.

*No memory = Number of failures caused by insufficient memory to create additional buffers.

The router does not create a new buffer if the number in free list equals the max allowed value.
If there is not enough memory in the router to create a new buffer, this is recorded as no memory.
If the number in free list is greater than the max allowed number, the router trims some excess buffers.
 

Area of concerns for a engineer is  number of  failures and no memory.Usually buffer failure occurs and the router corrects that..If the number of failures continues to increase, buffer tuning might be necessary. 

If the show buffers command output shows a large number of failures in no memory field, you must reduce the usage of the system buffers or increase the amount of shared or main memory (physical RAM) on the router.

Buffer leak and memory leak is another topic I want to mention here. Processor allocates a buffer but is not released after use. Same is with memory where a process is allocated with memory but does not release after it is executed. Both of them will occupy the memory but will never release which will cause low memory and might crash the device. Couple of symptoms I have come across is the device will not allow you to login as the SSH/telnet process doesn’t have enough memory, routing protocols misbehave, device shows up in CDP but is not reachable.

Show interface is the other command which helps to look at drops and buffer misses

Output queue 0/40, 1041 drops; input queue 0/75, 765 drops 
35252345 packets input, 547082589 bytes, 812 no buffer 

The input and output drops are due to the input and output queues being overrun by a burst of traffic. This is not related to a buffer problem, but rather to a process switching performance limitation.
No buffer  represents the number of packets dropped because there is no free buffer to copy the packet.

2.Implementing Hold Queue Limits

Hold queues are buffers used by each router interface to store outgoing /incoming packets. Use the hold-queue interface configuration command to increase the number of data packets queued before the router will drop packets. Cisco recommends increasing these queues by small increments (for instance, 25 percent) until you no longer see drops in the show interfaces output. Default output hold queue limit is 100 packets.

When you increase the number for an output hold queue, you might need to increase the number of system buffers.

Hold-queue command can be used generally in the following scenarios to avoid packet being dropped.

• You have an application that cannot tolerate drops, and the protocol is capable of tolerating longer delays.

• Low bandwidth or anticipated utilization is likely to sporadically exceed available bandwidth

3.Using Priority Queuing to Reduce Bottlenecks

Priority queuing is a list-based control mechanism that allows traffic to be prioritized on an interface-by-interface basis. Priority queuing involves two steps:

15-27

*Create a priority list by protocol type and level of priority.

*Assign the priority list to a specific interface.

Priority queuing automatically creates four hold queues of varying size. This overrides any hold queue specification included in your configuration.

Usually priority traffic is can be used under following scenarios:

• When the interface is slow, a variety of traffic types are being transmitted, and you want to improve terminal traffic performance

• If you have a serial link that is intermittently experiencing very heavy loads (such as file transfers occurring at specific times), and priority queuing will help select which types of traffic  should be discarded at high traffic periods

In general, I would start with the default number of queues when implementing priority queues. After enabling priority queuing, monitor  for output drops. If output drops are occurring in the traffic queue that has been specified to be high priority, increase the number of packets that can be queued using the queue-limit keyword option of the priority-list global configuration command . The default queue-limit arguments are 20 packets for the high-priority queue,40 for medium, 60 for normal, and 80 for low.

Further Reading:

Buffer Tuning 

 Understanding Buffer Misses and Failures

Troubleshooting buffer leak