News: Avaya announced their ONA Open Network Adapter

ONAAvaya has announced a new product wich is called ONA Open Network Adapter. It is an small form factor SDN Adapter wich is based on Open vSwitch. It is designed to bring plug-n-play connectivity for network devices. Basically Avaya is saying that you bring an ONA adapater for example to an branch office site and it connects over the WAN the remote location to your corporate network. The ONA can be integrated with the Fabric Attached technology into a SPB fabric or into the Avaya SDN FX controller architecture. Two models of the ONA are available. The ONA 1101GT provides one network and one device 10/100/1000Base-T port, and supports Power over Ethernet. The ONA 1208GT provides two network — 10/100/1000Base-T plus 100/1000Base-FX—and eight device 10/100/1000Base-T ports. The devices are passive cooled and have the form factor of a card deck. Pricing for the ONA will be start at $545. Avaya has scheduled the availability for mid-2015.

The Network Autobahn view:

The ONA is an interesting device, instead of managing remote offices or other external networks like seperate islands you can integrate these locations into your enterprise network and manage them with Fabric attach with the same comfort that you have in your SPB network. The ONA can be shipped to a remote location plugged in and deliver immediately connectivity for your services like voice or data no matter wich WAN service you are utilizing. That was done in the past with traditional WAN connections and VPN based products. Here you needed in most cases an on site engineer for the deployment, if that could work with the ONA plud-n-play it would be much more easy to deploy. An other interesting usecase would be temporary deployments that also can be connected over an OVA. In fact it is Open VSwitch based there will come more features to the device in the future. I highly appriciate that Avaya and Wind River have colloborated to contribute the Fabric Attach auto-attachment feature to the Open VSwitch community and making this feature widely available.

Fore more details check out this document from Avaya

http://www.avaya.com/usa/documents/avaya-open-networking-adapter-dn7702.pdf

Picture: credit by Avaya

Posted in All | Leave a comment

AVAYA VSP Software Update 4.1

AVAYA has released a new VOSS Software for their VSP8000 and VSP4000 Switches. Like most software updates it has some new features and bugfixes on board. The VOSS image that runs on several VSP devices is a Linux based operating system. The real news here is that AVAYA now is releasing an image across two different device families. The main goal is to have feature parity on the different VSP platforms that are running VOSS and brodcom silicon. So some of the features from the VSP8k are now available on the VSP4k and vice versa. From a development point of view it makes sense to bring a new feature to multiple platforms that using the same OS and the same silicon asics inside. I would also suggest that it is faster to develop and test a feature for multiple platforms instead inventing the wheel completly new for the different device families. As a customer I also like the idea to have the same feature set across different platforms, that makes deployments and planing more easy. Here is an overview what is new in SW 4.1.

VSP8000 SW 4.1
-IPv6 including static and dynamic routing
-IPv6 Access Control Lists (ACL)
-IPv6 Shortcuts
-IPv6 Routing between Layer 2 VSN
-Layer 3 VSN
-eBGP
-IP Multicast over SPB
-ISIS accept policies
-E-Tree and Private VLANs
-EAPoL IEEE 802.1x–2001
-MACsec
-Service Level Agreement Monitor SLAMon
-TACACS+
-VLACP statistics

 

For more informations check out the release notes.

https://downloads.avaya.com/css/P8/documents/101007558

https://downloads.avaya.com/css/P8/documents/101007562

The VSP 4000 has become besides the features mentioned above the SMLT and RSMLT with vIST feature. I already wrote about the vIST , if you are interested in that feature you should read the blog post: http://networkautobahn.com/2015/02/04/multichassis-link-aggregation-in-a-spb-fabric/ With 4.1 you get a lot of new options for a network deployment including the option to encrypt all uplinks with MACSec or deploying IPv6 over an SPB fabric. For MACSec and L3 VSN you need an extra license on both device families.

AVAYA hasn´t achieved full feature parity so far there are still 7 features that are not available on both platforms, but for me AVAYA is going in the right direction. Hopefully that effort will be continued.

 

Posted in All, Avaya | Leave a comment

Networking Field Day #NFD9

NFD-Logo-400x398Im looking forward to the next Networking Field Day that is starting next week. Stephen Forskett @SFoskett and the crew from Gestalt IT has thrown thogheter an imopressiv lineup. I really like the concept of the Tech Field Day events. The vendors that are brave enough to present their newest state of the art technology in front of the delegates can show up here, and you will always find direct compeditors in the linup like Cisco and Brocade or SolarWinds and NetBeez. The delegates are well known experts in the networking space from all the different corners of the technology and world. If one of the vendors has a hot technology these guys will ask the right questions and if the technology is not so impressive the delegates will give the vendors a hard time. Also part of concept is that they will listen to all the different vendor presantations in the short time of 3 days. That makes it easy to compare the different presentation with each other. Besides the interesting presentations that will be public available over the youtube channel from the tech field day event it is always a pleasure to read all the different blog posts and opinions that the delegates publish on their blogs. I wished sometimes all the delegates would come to my office for the next vendor presentation and be part of the Q&A session, that would be very helpfull.

On the Networking Field Day 9 we will see Brocade, Cisco, cloudgenix, cumulus, NEC, NetBeez, Pluribus Networks, Solarwinds and velocloud. That sounds like a quite excellent linup for me, looking forward to see what the vendors will show us.

Posted in All, Blog | Leave a comment

Multichassis Link Aggregation in a SPB Fabric

vsp_8kA decade ago Nortel introduced with the SMLT technology the first Multichassis Link Aggregation Technology. Today most of the major Vendors have a multichassis link aggregation technology in their portfolio like VSS / VPC from Cisco or IRF from HP. For me it was the end of spanning tree in the network core. A multichassis link aggregation provided the two main benifits of active active load balancing and fast failover times. At the beginning the technology was used a lot to connect switches with each other. It was also a very elegant way to connect servers to the network, you can provide additional bandwith to the servers and avoid a single point of failure in the network. In a datacenter, where all servers have 2 connections to two different switches with a multichassis link aggregation you can have a failure on one device or do a software update and everything is still running. Besides all the good points it comes with some limitations. Depending on the vendor and technology it is only possible to do a multichassis link aggregation across 2 or 4 devices. In most cases a lot of manuell configurations are needed to setup the links. And there was the limitations of a physical link between the two switches that shared the endpoint connection of your mutichassis link aggregation. The worst case here was when the direct connection between the two switchcluster member had a failure and you ended with a split brain situation, wich results in a complete network outage. Avaya has now introduced the virtual IST for their VSP8000 switches that elimantes the need of a physical connection between two switches that can be configured to form up a multichassis Link Aggregation or SMLT in Avaya terms.  When you have a SPB fabric based network it is only needed that the two switches are connected via SPB to the fabric. That provides extra flexibilty for your deployments as well as better redundancy.

vIST

Assuming you have already a SPB based configuration here are the steps that you need to configure a virtual Inter Switch Trunk (vIST)

First we need a vIST VLAN and an I-SID with a L3 interface

vlan create 4000 name "V-IST" type port-mstprstp 1
 vlan i-sid 4000 40004000
 interface Vlan 4000
 ip address 10.1.1.2 255.255.255.248 0

Now we configure the peer IP , wich is the IP from the partner device with that you would like to form up a virtual IST connection

virtual-ist peer-ip 10.1.1.3 vlan 4000

In the SPB config you also need a virtual bmac and the smlt peer system id from the partner device. Note with this extra virtual bmac you suck up one extra ID from your ISIS maximum devices. The limit was with SW 4.0.0.0 on the VSP8k 500 Backbone MACs.

spbm 1 smlt-virtual-bmac 00:00:00:00:10:ff
spbm 1 smlt-peer-system-id 0010.0001.0003

When you have configured the two pairs of your switchcluster and they have a connection to each other your vIST should be up and running

sho virtual-ist
================================================================================
 IST Info
 ================================================================================
 PEER-IP VLAN ENABLE IST
 ADDRESS ID IST STATUS
 --------------------------------------------------------------------------------
 10.1.1.3 4000 true up
 NEGOTIATED MASTER/
 DIALECT IST STATE SLAVE
 --------------------------------------------------------------------------------
 v4.0 Up Slave

 

To use a multichassis link aggregation / SMLT across your vIST you need to configure a MLT with type SMLT. Avaya has now 512 IDs available for SMLTs, so you configure always an MLT and gives it the attribute SMLT. The old SLT for Single Port SplitMultiLink Trunks are no longer needed. The configuration looks like that:

interface mlt 2
 smlt

The only thing that is important here is the MLT ID wich has to be the same on both peer nodes of your switchcluster.

At the moment the virtual Inter Switch Trunk is only supported on the VSP 8k, but I suggest we will see it in the near future on other Avaya VSP switches.

 

Posted in Avaya | 2 Comments

Logging and Beyond

I was burned many times after a faulty box has rebooted and all stored logs where lost.  For troubleshooting it is very important to have all relevant log data directly accessable. To have logs only stored on the productive devices ends in a troubleshooting nightmare. I have seen networks without a Syslog or NTP server, where it was a very time consuming job to find the cause for an outage in the log data. To have a powerfull log infrastructure is a must have in my opionion if you don´t wont to end in the debugging hell.

Sysloginfrastructure

Basic Logging Setup

As a general guidline for logging I recommand that all devices no matter waht it is send their logs to an external logserver. Mostly that is a syslog server but here it depnends on the divce type and vendor. The next step is that all devices should use a NTP server so that all loggs come into your logserver with the consistant time stamps. Try to collect as much as possible from your devices. You can apply later filters to reduce the amount of massages, but a massage that you never have received can be the missing hint in a truobleshooting hunt.

Logging Topology

Logging should be organized like a tree. Decentralized Logservers collect local massages and do a spoofed proxy forwarding to the next Logging server in the hirachy. Here it depends on your organization how you manage your logservers. Very common are the models of separtion by location or IT department. It also gives you a better handling if e.g. the Storage department has its own log server and use that for daily troubleshooting, when it comes to a problem that involves besides the storage devices also the network gear you can go to the central log server and see the massages from the storage and the network log server in one view. You can correlate the massages toghter and see how one event triggered an other event.  That helps a lot in finding complex issues in modern infrastrutures, where you have a high grade of integrations between the different components.

One format to rule them all

It is a challenge to get all the different masseges to one format that you can analyse. It starts with the different Log formats, syslog is the most common, but there are some vendors that do their own protocolls like Checkpoint. Here you need a log server that can collect these different input protocolls and formats and parse them into a standard.  It is very frustrating when you have to browse between 5 different tools and formats to view logs for an incedent that you are investigating. Besides the classic log data you have on most network gear also flow data from IPFIX; Netflow, JFlow, SFlow … available. It can be very usefull to have the ability to corelate the logs and the flow data together especially for IT forensics.

The right View for your Logs

In the past we had only the command line to view the log data. But when you you do advanced analysis of log data a graphical interface becomes really handy. Open source projects like greylog2 have a powerfull web interfaces that brings clever extra features on the table. It is possible to build a complete Logging infrsatructures with open source. At the end of the logging hirachy it makes sense to have an SIEM solution in place wich has more capabilitys than a normal syslog server. Besides the security aspect a SIEM system can also be used for troubleshooting network problems. There are not many open Source SIEM systems available, I am only aware  of Alien Vault OSSIM, wich is limited to one user and the free version of Splunk wich is limited to 500MB/Logdata per day. For testing in the lab both are a good option and on both solutions are full feature commercial versions available. Another usefull addon for logs is a visulization engine. Sometimes one graph can show you more than 1000 log entrys. Very interesting here is the commercial solution from ArcSight.

For further Informations about Logging I can also recommand the packet pushers show 192 Logging Desgn & best Practice. In the show the packet pushers hosts had a discussen about Logging with Jay Swan, Wes Kennedy and greylog develoopper Lennart Koopman.

http://packetpushers.net/show-192-logging-design-best-practices/

Posted in All, Monitoring | 3 Comments

Network speed and VM NICs

I stumbled upon an interestiutilizationng fact about the speed of VM NICs. In general I suggest that the maximum bandwith that a NIC can send or receive is limited on the interface speed. So an Intel Pro/1000 Interface NIC on an physical Server can only send with 1Gb/s. It looks like that this satement is not true for VM NICs.

I was surprised as I did some testing on a VM and saw that it was possible to transmit ~9 Gb/s on a 1Gb/s Intel Pro/1000 VM NIC.  In my setup I had 2 VMs running on 2 ESX hosts on different physical servers. Each physical server had a 10 Gb/s NIC. On the VM was a Windows server running that showed a 1Gb/s connection that was running with a 99% utilization during the test. I thought that you need a vmnic3 driver installed to get 10 Gb/s to a VM , looks like that is not the case. The para virtulized driver for the NIC only emulates the speed for the OS. So you can have up to 10Gb/s on a 1Gb/s VM NIC.

Posted in All, Blog | Leave a comment

Connectivity Fault Management for SPBM

duct_taped_fibreIEEE 802.1ag CFM or Connectivity Fault Management is very popular in the provider space to troubleshoot MPLS networks. Avaya has implemented CFM for all their switches that runs SPBM. We are facing here the same problem that we have in troubleshooting MPLS networks, we need to figure out wich path across a given network the traffic is forwarded. In SPBM you can have multiple redundant equal cost multipath wich will be presented for a standard trace from a connceted host like a single hop. CFM gives you the ability to run Layer2 pings and traceroutes across your SPBM fabric.

To setup CFM for Avaya switches you need this command:

cfm spbm enable

I would recammand to enable cfm on all devices in a SPB network. Now you can use the L2 ping and trace commands. You need the SBM B-VLAN as a paramater. Here it can be that you have an other path across your network for the second b-vlan. The routername is the sysname of the switch, here it is an good idea to have something human readable that makes troubleshooting more easy. Note the routernodename is case sensitive.

ERS8k:5# l2 ping vlan 4051 routernodename VSP7k
 Please wait for l2ping to complete or press any key to abort

----00:10:00:32:00:00 L2 PING Statistics---- 0(64) bytes of data
1 packets transmitted, 1 packets received, 0.00% packet loss
 round-trip (us) min/max/ave/stdv = 1765/1765/1765.00/ 0.00


ERS8k:5# l2 traceroute vlan 4051 routernodename VSP7k

 Please wait for l2traceroute to complete or press any key to abort 

l2traceroute to VSP7k  (00:10:00:32:00:00),  vlan 4051
0    ERS8k              (00:10:00:11:00:00)
1    VSP8k              (00:10:00:12:00:00)
2    VSP7k              (00:10:00:32:00:00)


ERS-11:5#l2 traceroute vlan 4052 routernodename VSP7k

 Please wait for l2traceroute to complete or press any key to abort 

l2traceroute to VSP7k  (00:10:00:24:00:00),  vlan 4052
0    ERS8k              (00:10:00:11:00:00)
1    ERS48              (00:10:00:21:00:00)
2    VSP7k              (00:10:00:24:00:00)

I hope that Avaya will implement also in future Continuity Check Meassage CCM, Delay Measurement DM and Y.1731 for performance monitoring like jitter, latency and frame loss.

Posted in All, Avaya, Howto | Leave a comment

8 Rules for Network Autobahn Engineers

To run a network can be a har8Rulesd challenge sometimes. It helps a lot when you stick to some generell rules and apply these to all the work and devices for that you are responsible. I have seen networks that had no standards at all, every device was configured somehow in a different way. This ends in an unmangeable desaster, where you can not predict anything. I prefer that you have one standard that is appllied to all devices, it need to be consistant, so that you have a stable and predictable network. Besides the techniacal challanges a network can only run as good as the Engineer has designed and installed it. So the main goal is to have the complete package.

1: Make standards where ever you can

Not everything can be equal configured on all devices thats for sure. It is depending on the device type, site or software version. Where ever you can make standards that can be applied over all your devices try to do it. For me it is important that general paramater like NTP Server, Syslog Server, SNMP Trap Reveiver etc. are configured identically. So that you have the same time on all devices, and all the logs/traps on one central log server , that helps a lot when you have to troubleshoot a network problem.  Also very important is that you have the same protocols and versions on all devices. To have standard STP , PVSTP, MSTP and RSTP all configured in the same network does not make sense at all. A very difficult aspect is the sofware version that runs on your devices. I prefer that it is the same on all devices. We all know that most software releases have bugs inside, if you have on all devices the same software you know at least these bugs. In an given network where you have 20 devices with 20 different versions of software code inside it can be pretty hard to predict what is going on. A second point here is that often tiny little details have changed like CLI commands or SNMP MIBs.

2: Documentation

The goal is a valid up to date documetation of the entire network. That isn´t the case in most real world deployments. As an external consultant you end up often in the situation that you have no documentation at all from a network and have to start discovering the network as the first task. I try to achieve a good level of documentation with a simple policy , no changes without documentation. In the process of the change like e.g. adding a new switch to the network, it has to be part of the workflow to update the documentation. If it is not done at the installtion date it is in most cases never done.

3: Monitoring

All devices that are in production have to be monitored.  If something brakes that is not monitored you will get calls very soon and have no clue of what is going on. Here it is also a good idea to implement monitoring in the workflow. When a new device is added to the network, the monitoring should be setup within this process. I have always a better feeling for the network when I know what is going on. With a good monitoring in the backround you know what is going on before somebody calls you. When you can react before something breaks the monitoring is working perfect.

4: Testing

Do extensive testing of all new stuff before you implement it in a production enviroment.  New stuff can mean Device types, protocols, network designs, software code…. or other thinks that you haven´t seen or tested before. Not everthinks works in the real world exactly the same way that it is discribed by the vendor documentation. Rebuild as much as possible in the lab and try to break it. Make your expierences and get comfortable with the technology. If it does not work in the Lab you should not role it out to your production network.

5: Educate yourself

In IT time is always running. Try to keep up with new technologys and trends. Sometimes a new technology or feature can solve a problem that bothered you for a long time. To invest time in research can lead you into great new opportunitys.

6: be Open Minded

Sometimes I am stucked with an particular problem and have no idea how I can fix it. Here it helps to think out of the box and discover new ways to solve problems. Keep contact to other IT departments, sometimes the server or storage guys can give you input from an other point of view that leads to a solution of a problem that you haven´t thought of before.

7: Chose your weapons wisely

You will need a wide range of tools for all the different task in your daily work. Sometimes it makes sense to use a swiss army knife tool that can do a lot of different tasks and sometimes you need a much more specialised tool that is disgned only for a particular task. Here it is important to know wich tool you need for a task. I also try to review my tools on regular bases, sometimes you find a new tool that gives you some extra features.

7: Never stop Optimizing

Don´t get stuck with an configuration or protocoll that is outdeted. Only because we had used something in the past for a long time does not mean that this it is still the best way to do it today. Technolgy is evolving every day , leverage this in your advantage. An good example is HSRP. On most Cisco devices are 3 first hop redundancy protocolls supported: HSRP, VRRP and GLBP. VRRP has the benefit that it is an open source standard and GLBP gives you active-active load balancing over multiple L3 routers.  In real world deployments I still see that HSRP is used in most networks, wich is not the best choice in my opinion. Don´t be scared about Changes, when you get a benefit from a change, test it, roll it out and have a better system at the end of the day.

8: Share your Knowledge

If you have solved a problem , share your expierence with your co-workers and the network community. A post in a network related forum is done in a few minutes and can help someoneelse with the same problem.

 

 

 

Posted in All | Leave a comment