There is a bug in the tg3 driver on the ESXi hosts (1gbit broadcom cards in the new hosts). If the network card is put under load and netqueue is enabled it will sometimes decide to drop all traffic. Essentially i’ve disabled netqueue and the problems have gone away…. as per this vm kb :
http://kb.vmware.com/kb/2035701
The isues will present themselves as log entires like so;
2012-11-19T18:58:52.137Z cpu17:4155)<6>tg3 : vmnic8: RX NetQ allocated on 1
2012-11-19T18:58:52.138Z cpu17:4155)<6>tg3 : vmnic8: NetQ set RX Filter: 1 [00:50:56:71:46:87 0]
2012-11-19T18:58:52.138Z cpu17:4155)<6>tg3 : vmnic7: RX NetQ allocated on 1
2012-11-19T18:58:52.138Z cpu17:4155)<6>tg3 : vmnic7: NetQ set RX Filter: 1 [00:50:56:71:46:87 0]
2012-11-19T18:59:12.139Z cpu21:4155)<6>tg3 : vmnic4: NetQ remove RX filter: 1
2012-11-19T18:59:12.139Z cpu21:4155)<6>tg3 : vmnic4: Free NetQ RX Queue: 1
2012-11-19T18:59:22.137Z cpu24:4155)<6>tg3 : vmnic4: RX NetQ allocated on 1
2012-11-19T18:59:22.138Z cpu24:4155)<6>tg3 : vmnic4: NetQ set RX Filter: 1 [00:50:56:71:46:87 0]
2012-11-19T18:59:42.138Z cpu21:4155)<6>tg3 : vmnic7: NetQ remove RX filter: 1
2012-11-19T18:59:42.138Z cpu21:4155)<6>tg3 : vmnic7: Free NetQ RX Queue: 1
2012-11-19T18:59:42.140Z cpu21:4155)<6>tg3 : vmnic4: NetQ remove RX filter: 1
2012-11-19T18:59:42.140Z cpu21:4155)<6>tg3 : vmnic4: Free NetQ RX Queue: 1
2012-11-19T19:00:02.139Z cpu28:4155)<6>tg3 : vmnic8: NetQ remove RX filter: 1