Skyport Systems @ NFD11

skyportSkyport Sytems has used the NFD11 to show the first time their new innovativ product.
Skyport is a startup company with a very unique product that brings a new layer of security to the table.
Basicly you can verify that a server is not compromised for e.g. by a rootkits. The problem is to verify that a server is not compromised these days. To survive a wipe/installtion process, rootkids are placed in HW componetnts like e.g. the Firmware of HardDisks and SSDs. Even with a fresh OS installation the server would be immediatly compromised again. At the moment it is hard to address this attack vector. Skyport positioned their product as an extra layer of security for the high mission critical appliactions. How does it work ? Skyport has shown a Hardware based NIC that has TPM Chips, CPUs and RAM. So all the hardware and firmware inside of a server can by verified and inspected.
It is also possible to control and manipulate the traffic that goes across the Skyport hardware. Besides Firewall functionsalitys you have with Shield WEB Application a Crypto Creditional Proxy that can act as an encryption break up point so that SSL encrypted data can be inspected here as well.
You also get new Logging capabilities for your Skyport protected servers. Of course you can run a Virtualization Hypervisor on a Skyport protected Server.

Network Autobahn View:
We have to protect a System against attackers in all possible ways. Skyport adds an additional layer of security and addresses an attack vector that is not covered by any other classic security solution that I am aware of.
Will we deploy a Skyport NIC to all our servers ? Maybee not, but for the buissness critical application I see a pretty sharp usecase.

For more informations check out the Video from the NFD11 Skyport presentation:

Posted in All | Leave a comment

Network Field Day 11 Aftermath

NFD-Logo-400x398I have attended to Networking Field Day 11 in San Jose California wich is organized by GestaltIT.
It was a buisy week with a lot of state of the art tech. I had the pleasure to see from a wide range of vendors their products and visions. Some vendors had their first preseation at NFD and other are regular contributer. In my view the Network Field Day event is a win win situation for everybody. The vendors can present their newest innovations to the right audience insteat of an oldschool broadcast approach of merketing.
This is the introducing to a series of blog posts about the presnations that I have seen at NFD11.
I will give you fo all preenations a quick overview and my opinion as well. If that caught your interest check out the TechFiled Youtube Channel
and watch the complete presnation.

Network Field Day 11 Videos

Posted in All | Leave a comment

Happy New Year in IT

2016When the year is ending and everybody is thinking about XMAS in a lot of  IT departments  it starts to get hectic. There is some IT budget left and it has to be spended before the year is ending. I have seen this in many businesses and it is especially true for government organisations. So like at home where I have to assemble some Lego Toys for my children on XMAS there are delivered a lot of large packages at the office that have to be unboxed and rolled out into production before the year is ending. So not for all IT folks the end of the year is as relaxed as it should be. On the first weeks of the new Year I had often to do a lot of clean up work to get everything right for that was no time during the installation like monitoring and documentation.

Hopefully you all have finished your 2015 projects in time and stayed at home with your family at XMAS instead of being busy in the office.

Happy New Year 2016.



Posted in All | 2 Comments

Network Field Day #NFD11

NFD-Logo-400x398ve been invited to attend the Gestalt IT´s Network Field Day 11 held in the Silcon
Valley on January 20th – 24th 2016.

For those of you that haven´t heard of the Tech Field Day events so far. The idea behind Network Field Day is to bring together a bunch of delegates that will attend one week of presentations by different top IT vendors. All the presentations will be streamed live and uploaded to youtube after the event. That includes Q&A sessions after the presentations. It helped me a lot to listen to the Tech Field Day presentations before scheduling an appointment with a possible vendor.

I am honored to be part of the Tech Field Day 11  and am looking forward to meet up with the other delegates:
Ethan Banks
Greg Ferro
Ivan Pepelnjak
John Herbert
Jon Langemak
Brandon Carroll
Terry Slattery
Jason Edelman
Matt Oswald
Jordan Martin
Michael McNamara


You can follow Tech Field Day 11 on Twitter @NFD11

The following disclosure also needs to be made – my travel and lodging costs will be covered by Gestalt IT and the sponsors of Tech Field Day11.

NFD11 Presenting Sponsors












Posted in All | 2 Comments

End of Sale Avaya ERS8000

ERS8000EoSAvaya has recently published an end of sales notice for the ERS8000 product line. The ERS8000 was introduced as Passport 8000 in the year 2000. The product is now for nearly 16 years available. I have configured a lot of new technologys the first time on the Passport/ERS8k. For me the 2 most amazing features that have been introduced on this platform the first time was SMLT and SPBm. The SMLT SwitchCluster features wich was introduced in 2001 was the first Multi Chassis Link Aggregation technology. SMLT was in 2001 a real cutting edge technology that was ahaed of most of the compeditors. For example Cisco introduced with VSS their Multi Chassis Link Aggregation technology in 2008 wich is for IT standards ages later. 10 years later the second next generation technology SPBm was introduced. In 2011 the first SPBm pre standard implemenation was showing up on the ERS8800 platform.

So it is time to say goodbye to the Passport/ERS8000. At the end of the day the complete industry is shifting to Linux based switching OS and the old monolithic OS based switches are fading away.

Some of the ERS8k developpers from Avaya have created a Goodbye ODE , wich I have seen recently on a Avaya presenation:

“When we first turned you on SMLT was quiet new
We had some tough times but we made it through
Alone in the rack looking naked and small
Before we knew it ERS modules populated all
Bandwith demands came quick and came swift
When we gave you E modules you just wouldn´t quit
Who would of thought 10Gig to come fast
Your poor little E modules just wouldn´t last
When R modules came so did netflow
You got super mezz cards but had problems below
Slot 10 was tired and couldn´t keep up
So your body was replaced and you were brand new pup
they lauded and loved you and gave you a new name
8800 they said but you were still the same
Ten days before retirement a power supply quit
We knew at the time we had to be quick
After the more than a decade you served us well
Oh the good times we had and stories we tell
Your out of commision but you still stand tall
Your performance and relaibilitywill be remebered by all”

Here is the Link to the EoS Notice:

Posted in All | Leave a comment

MicroBursts A Troubleshooting Nightmare

One of the most difficult problems to troubleshoot in a Network are Microbursts. This is a really though one. So what is actually the problem with Microburts ? You have a Traffic peak in the network that is only present for a subsecond. Sometimes these spikes can fill up a 10Gigabit Interface at full line rate. The result is that you have typically on multiple devices in a VLAN/Subnet a high rate of TCP retransmits and resets wich causes ~25% Packet Loss. In most cases the server / appliaction teams detects first performance problems that occur sporadically. When this is reported to the network team it is nerly invinceble on the network side. The normal sources for statistics and troubleshooting will show up nothing. For example the monitoring server that polls e.g. every minute the Interface statistics will show up nothing. Also the show commands in the CLI shows on most vendors a statistic over a timeperiod of 10 seconds, wich will round down the burst that was only present for a subsecond. So it looks like that there is no problem in the network. To find the problem it helps to have some sniffer traces during a Microburts that show the TCP Retransmits and Resets. At this time you have to think in a different direction to hunt down the micoburst. Depending on the switch vendor you have to look at a different error counter. The root problem here is that an asic reaches the maximum of throuput and starts to drop packets. If you are lucky you have a counter for that drops like “Drops on no Ressources”.

What can you do to resolve the Problem ?

On the server / application side it is possible to change the traffic profile to remove the burtsy behaviour. That is really hard to achieve and can only be done with apllications that you can change and control. If you can do that this will resolve the issue with Microbursts for one type of Server / Application. You have to be aware that you can run in the same problem again in that network when you deploy for example a new application.

The other method to avoid the problem is to split up the uplinks that are connected to severs that show the bursty behaviour to different devices or asics. It also helps to have more bandwith on the uplinks available than the burst could fill up. So when the Microburst spikes up to 10Gig a 25 or 40Gig Uplink also resolves the issue.

Sometimes you have Micobursts sporadically in a network for years undetected. With strange performnce tickets that are unssolved for a long long time. This is really hard to detect , so keep Microbursts in mind for the case you are dealing with this kind of problems.


Posted in All | 2 Comments

Switch OS Reverse Engeneering

It is really hard to get informations about the proprietary OS that runs on many switches. The vendors don´t give away many informations how it actually works under the hood. The old model of security by obscurity is still applied here. I saw on the 25C3 conference in Berlin the “Cisco IOS attack and defense” talk from Felix FX Lindner that changed my mindset about code quality inside of switch OS completly. Felix FX Linder reverse engeneered the IOS code and showed very detailed how IOS works and wich attack vectors can be leveraged to get control over an IOS based device. Felix is one of the most talented persons in the community when it comes to reverse engeneering and I am very thankful for all the time and effort that he has spend on this project. The talk is about 1 hour and covers a really deep dive into Ciscos IOS code. I learned more about how IOS works from this talk than on all presantations that I have ever seen from Cisco.

This talk is from 2008 and was the first of a series of switch OS reverse engeneering projects from FX. The next target was the Huawei VRP OS. The results FX presented on DEFCON 2012. Huawei had a joint a venture with HP and I it looks like that most of the results are also apply for the H3C devices from HP. The myth that Huawei has copied the IOS code was disproved by FX. He found out that the Huawei VRP OS is based on VxWorks. At the end of the talk his devastating summary is “90´s style bugs, 90´s style exploration, 0 operating system hardening … no security advisories..”.

Beyond the physical switches FX also reverse engeneered the Cisco Nexus 1000v virtual switch. In the talk “Cisco in the sky with diamonds” FX presented the results of that research at the Signit 2013 conference.The NX-OS based Nexus 1000v is based on a Montavista Linux that runs a 2.6.10 Kernel. FX und Greg found a jailbreak wich they show in the talk and mention that the same jailbreak also works on the physical Nexus devices.

This shows the level of security that is embedded inside of the switches that FX has investigated is very poor. I think very different since I am aware of the resaerch of FX when it comes to protect a switch from getting owned by a hack. It also explains a lot of the bugs that I have expierenced in the past. Hopefully FX and Greg will continue their excellent work in the future.




Posted in All, Blog | Leave a comment

Packet Pushers Podcast Show 250 – How To Document A Network

PPI-Weekly-New-330x330-optI have recently attended to the packet pushers podcast show 250 – How To Document A Network with the packet pushers hosts Ethan @ecbanks and Greg @etherealmind. It was the 3rd time that I have attended to the packet pushers podcast. We had an interesting discussion that gives a pretty good overview about the most important topics regarding to network documentation and documentation tools. We nearly hit the 90 minutes mark. The show can be downloaded here:

If we missed your favorite documentation tool, feel free to leave a comment.


Posted in All, Blog | Leave a comment