Throttling VMware Site Recovery Manager 5

In my day job as a VMware Architect I often come across strange problems which require some internet based research to solve. I generally assume that if I have a problem there’s a good chance that somewhere, someone else will have experienced the same or a similar problem and documented the solution. In this case I found almost nothing online so here’s a writeup!

I’ve been building a small SRM 5 implementation (licensed for 75 VMs) for a client who use NetApp FAS at both the primary and recovery sites with SnapMirror to replicate the data. The client have Fibre Channel disk at the primary site replicating to a single filer head with  SATA disk attached at the DR site (with some additional fibre channel disk for VMs that are always powered on). When we first ran a simulated test we found that after the first 5 or 6 VMs had powered on the majority of the remainder experienced timeouts waiting for VMware Tools to start and looking at the VM console it was obvious that the VMs were hitting a disk bottleneck. The actual cause of the disk bottleneck wasn’t quite as simple as the fact that we were using SATA disk to recover from (this could form a completely separate writeup on the joys of deswizzling) but it was a big part of the problem.

The client has an RTO (Recovery Time Objective) of 24 hours and SRM was completing the recovery plan within 1 hour. Even considering that there is additional work to fit into that 24 hour period it would be perfectly acceptable for SRM to take up to 10 hours to recover the VMs. While SRM was only taking 1 hour to recover the VMs we were seeing failed services in the Windows OS on the VMs and, since the task that waits for VMware tools was timing out after 300 seconds, VMs in subsequent priority groups were being powered on before their pre-requisite VMs were fully booted.

Continue reading “Throttling VMware Site Recovery Manager 5”

Muang Thai, Brighton

Muang Thai is situation near Brighton’s remaining pier, one road back from the seafront. From the outside Muang Thai looks like any other pleasant but fairly generic Thai restaurant. Inside, the decor is clean and light but very obviously a Thai restaurant. My meal consisted of a green papaya salad and a vegetable Thai Green curry with steamed rice all washed down with a couple if glasses of the house red wine.

Continue reading “Muang Thai, Brighton”

Day 2 – Sights, Sounds and Saris

First a photo:

The Streets of Bangalore
The Streets of Bangalore

Day 2 is our formal kick off. Formal because we sat in a circle and introduced ourselves. We all introduced ourselves in turn. I’m used to the standard “Hi I’m Chris and I work in ITS, Server SPL, Microsoft and VMware delivery”. This time we all spoke about our background and how we ended up on this adventure. I was amazed to discover that everyone in the group had a fascinating story to tell. I’ve rarely met a group of people who are so open and honest amongst a group of virtual strangers. This could have easily been a stage for the more senior members of the group to talk about themselves at length, we are no stranger to self promotion. However everyone seemed keen to listen to everyone else and nobody used this as an opportunity to show off.

Continue reading “Day 2 – Sights, Sounds and Saris”

Bangalore or Bust

The following is my post from the official CSC blog:

Firstly, a quick introduction. My name is Chris Greenwood. I am an IBM employee based in London, England and this is the very first post from a journey that promises to be a great adventure.

I was lucky enough to be selected to participate in IBM’s Corporate Service Corps. I was assigned to the ‘India 9’ team (or Blue Tigers as we are commonly known) along with 11 others from 9 different countries. Unfortunately before we even left a broken ankle reduced our team of 12 to 11.

It’s now the end of my first day. I’m sitting in my hotel on the roof terrace. I’d like to say it’s peaceful but I’m surrounded by the noise of cars, rickshaws and bikes and the incessant sound of horns that seem to permeate everywhere you go. What follows is the story of my first 24 hours…..

Continue reading “Bangalore or Bust”

Off to Bangalore

So I’m sitting at gate 20 in Heathrow terminal 4 at the crack of dawn waiting for a flight to Paris. The gate isn’t open yet and won’t be another 10 mind yet there is a queue forming in front of the desk. Why are these people in such a hurry to get on a plane? Do they think the seating is unreserved? (its not). Do they think they’ll get the Paris faster by standing 10 meters closer to the gate that my semi-comfy chair? These both seem unlikely, maybe they’ve realised that they’ll be sitting down for the next hour or so and they’re enjoying their last few minutes of Standing freedom.

I’m now on the plane I managed to get an emergency exit seat so I have about 2m of leg room but no window. Sounds like a fair compromise to me.

Think I better send this and turn off my phone. The cabin attendant just decided to remove my laptop bag from the overhead locker, drop it on the seat then move it somewhere. I said don’t worry its just my laptop but I don’t think it crossed the language barrier.

I’ll return later and explain the title of this post!

Joshua Radin at Shepherds Bush Empire – with a surprise appearance by Zach Braff

So I’ve just returned from a Joshua Radin gig at the Shepherd’s Bush Empire. To be honest I’ve never seen him live before and was slightly concerned that we might all be compelled to slit our wrists well before the encore. I was pleasantly surprised. The set was a great mix of some newer more upbeat stuff and as Joshua himself described it “the soundtrack to your suicide”. To be honest I know the older stuff much better and his performance was perfect. I’ve never been to a gig where the artist spends so much time talking between songs but to be honest I guess many of them don’t have a lot to say. Joshua Radin was actually interesting to listen to and came across as a regular guy who’d come to play us some music.

So just before Joshua Radin came on stage I was just telling Alex that my good friend Olly went to one of his gigs in New York and saw Zach Braff. She looked up and there he was up on the balcony at the front directly above and behind us less than 10m away. He’d been keeping a fairly low profile in a flat cap but moments later everyone spotted him and he eventually took it off and waved at everone below. Later on in the evening at a quiet point Zach shouted “I love you” and Joshua responded “That’s very manly Zach”. Joshua Radin eventually introduced him near the end and explained he was over in London for his birthday.

Anyway a great evening made even more memorable!

What I’m listening to – Xfm’s Songs of the Decade

I’m listening to the Xfm Songs of the Decade while stuck in a data centre. Its so noisy in here that I need some musical relief. Anyway I decided to add the top 100 to a Spotify playlist for future listening. Unfortunately 10 of them aren’t on Spotify so here’s 90 of the top 100 songs of the decade!

Xfm’s Top 100 Songs of the Decade

These are in 100-1 order and the following tracks are missing:

98. Feeder – Buck Rogers
92. Oasis – Little by Little
80. Red Hot Chili Peppers – Can’t Stop
62. Arcade Fire – Rebellion
56. Arctic Monkeys – Fake Tales of San Francisco
48. Oasis – The Importance of Being Idle
39. Oasis – Stop Crying Your Heart Out
28. Arctic Monkeys – Mardy Bum
5. Elbow – One Day Like This
2. Arctic Monkeys – I Bet You Look Good on the Dancefloor

I love Spotify but none of the above are obscure hence their inclusion in the above list. Oh well I

While I’m at it. Here’s what I’m listening to at the moment, mostly newish stuff but some older tracks I rediscovered on Spotify too!

Let’s Dance to Joy Division

Update: Found a few more but still only have 93 – looks like a couple disappeared!

Hue

We arrived in Hue and checked into our hotel, Binh Duong 1. We had been offered a basic double room for $8 but decided to splash out and go for the $12 room that contained a computer with internet access as I was getting very far behind on this blog. The room was on the top floor of the hotel, was very clean, well equipped and had a great view. We left the other six of our group downstairs to decide where to stay, giving them our room number so they could get in touch. About an hour later we had a call on the room phone from three of the guys who had decided to hire bikes for an hour for a quick ride around the citadel. We decided to join them and paid for a half days bike rental though this only came to 15p! Our hotel and pretty much all hotels and restaurants are located on the south side of the river in new or European city and the main city itself is across the river. The entire old city on the north side is enclosed by a 20m thick wall that run around the city in a square with each side of the square being 2.5km long. We cycled across the bridge and through one of the gates into the old city. Inside the old city is 2 further walled areas. The first that you reach is the imperial citadel which would have been used by the mandarians, civil and military noblemen as well as everyone who served the emporer. Inside the imperial citadel is a further enclosure called the forbidden purple city. This area would have been used solely by the emporer, his wives and concubines. In fact the only men allowed inside the forbidden purple city who weren’t related to the emporer were eunuchs.

Continue reading “Hue”

The Demilitarised Zone (DMZ)

So a minibus eventually picked us up from our hotel around 30-40 minutes late. We were one of the last to be picked up so they were probably delayed by a few people on the way. We were taken to the sleepy bus which is a full sized coach fitted with rows of nearly flat bunk beds which convert into seats. We got on the bus and found ourselves a couple of top bunk beds next to each other near the back of the coach. The beds around us filled up with other people many of whom were on their own and within a few minutes we all got talking, sharing stories of travels. We talked for a few hours an eventually the lights were turned off and we went to sleep. Nobody slept particularly well but I did at least get a few hours sleep. We were woken up at 6:00am by traditional Vietnamese music and a video on the TV. Soon after we stopped for breakfast at Dong Ha. While eating we were approached by a guy who offered us a DMZ tour. We had planned to tour the DMZ from Hue but looking at the map we could see that Dong Ha was very close to the DMZ and Hue was another two hours down the coast so we’d have an extra four hours travelling time to see the DMZ from Hue. Before we got on the bus I’d had an e-mail from Tom saying that they had done a really good DMZ tour from Dong Ha with a guy who’d approached them at their breakfast stop. When I mentioned the names Tom and Craig  to the guy he beamed with delight and repeated Tom and Craig, it was the same guy who they’d taken the tour with. I knew he probably wasn’t just making this up as he could actually pronounce Craig, something almost nobody here can (including the Americans we have met). We were due to be picked up by our hotel from the bus in Hue but the restaurant owner offered to call and cancel the pickup so we agreed and along with another of the giys we had met we decided to do the tour. Alex and the other guy, Luke got back on the bus to get our stuff and at the same time persuaded another five people to join us. We unloaded our stuff from the bus and the eight of us watched our bus to Hue drive off. The tour guide was called Hoa and spoke pretty good English. He gave us about an hour to have some more food and wash, change etc and then when we were ready all eight of us piled into a minibus. The first stop of the tour was the former US Con Theiu firebase. While there was not actually much to see at the firebase apart from one concrete bunker what was so amazing to see was the vegetation in the DMZ compared to the surrounding areas. During the war the US napalmed the entire south side of the DMZ to prevent the Viet Cong hiding in the dense forest. This conbined with the agent orange that they sprayed completely destroyed all plant life and even now over thirty years later the vegetation still looks very young an unestablished despite massive government planting schemes. The concrete bunker at this firebase was one of the only ones that was made of concrete and was reserved for the high ranking officers. Everything else was constructed from sandbags and our guide picked found a few partially filled sandbags from the thousands that litter the area. The area that we were in has been cleared of mines by Australian, French and British mine clearing teams but our guide jumped into a pit and picked up a couple of obviously preprepared mine detonators that had been found in the area by locals. Evn though we knew the area was relatively safe we still followed the guide’s path exactly and didn’t stay from the tourist trail. The bunker was pretty interesting, pockmarked with bullet holes and with the word California scratched into the concrete by a soldier during the war. After seeing the bunker we headed back to the minibus and drove on to a North Vietnamese cemetary where over 10,000 north vietnamese troops were reburied after the war. Seeing the endless rows of graves was a pretty sobering experience as you can see from the photo that I have put up in the DMZ gallery. Throughout the day while driving around in the minibus we would often stop to looked at bomb craters left in the land. Some were as big as 15m wide and the guide told us that they had been as much as 10m deep though were only about 8m deep now due to erosion. We also stopped at one of the small pockets of natural jungle that hadn’t been too damaged by the bombing which was a stark contrast to the rest of the scenery and even this was filled with massive craters. We were told that in the area just north of the DMZ 7 tonnes of explosives were dropped for every person living in the area. After these sobering facts we headed over to the Ben Hai river which formed the border between North and South Vietnam. We were taken to, what was at the time the only bridge over the river which was fenced off when Vietnam was divided into North and South. The bridge was a reconstruction of the original for tourists and as a memorial as the orginal was blown up by the US during the war. We were told an amusing side story about how a mini war was fought across the river with boths sides competing to have the tallest flag and the loudest speakers spouting propaganda across the river. Our next stop was the Vinh Moc tunnels. These tunnels were one of over 100 tunnel networks built by the local population as the only way to escape the bombing. The tunnels are 23m underground at thier deepest and comprised of three stories of tunnels including a meeting area, a maternity ward (where 17 babies were born, all of which apparaently survived the war), kitchens and tiny rooms where whole families spent months at a time. The tunnel network had not been enlarged for westerners as it was built bigger than the Cu Chi tunnels down south but there were very few places where I could stand up and in most areas had to walk in a seriously hunched up position to avoid banging my head on the hardened clay roof. We saw rooms on all three levels and in total spent about 45mins to an hour underground with one break outside at one of the seventeen exits. You don’t really realise how deep underground you are until you reach a ventilation shaft and look up, which makes you very glad that there are backup exits! The tunnels did have electric lights in some places but these had been put in to recreate the dim lighting conditions that the original paraffin lamps had provided and a torch was still necessary in most areas. Luckily I was given the second torch with the guide carrying the first which made the whole experience a little easier. Nevertheless after almost an hour we were both glad to see daylight again and wouldn’t have wanted to spend much longer underground. Our last stop was the partial remains of a rusted US tank on the side of the road at what had been another US firebase. There is very little to see at any of the old US bases as what was left over after the war including the many of the mines and unexploded ordnance was scavenged by local scrap metal sellers causing hundreds of casualties.

We headed back in the minibus to the cafe where we’d started the day for some much needed lunch and, when we were finished we climbed aboard the same minbus for the trip to Hue. We left the guide in Dong Ha and he instructed the driver to take us to Binh Duong, the hotel we were booked into in Hue. Hearing that we were booked in, the other six of our part decided to stop there to check it out. The ride to Hue was an interesting one with the second person on the bus trying to persuade Vietnamese people on the roadside to climb into our already full minibus. Luckily for us she was mostly unsuccessful.

When we got to Hue I followed where we were going on the map as I usually do so that I know were we are and sure enough not far from our intended hotel we veered off the route that I thought we’d take and stopped at another, shabby looking hotel. We were told to get off and assured this was our hotel. When I explained that I knew that this wasn’t the hotel we were told that there were police over there and that the minibus couldn’t take us any further! Of course the owner of this hotel immediately came out offering us seemingly excellent rates and a couple of our group were tempted but I said that I wouldn’t consider anywhere that I was dropped against my wishes as the minibus owners had obviously pulled this one on us to earn commision. I could see from the map that we were no more than 15 minutes walk from the hotel we had booked so I lead our group over to our hotel on foot. On the way we were hassled by a guy who wanted us to stay with him but when he realised I was set on the hotel I had booked he fetched an employee of our hotel who led us to the right place. It turned out that the cafe owner in Dong Ha hadn’t phoned our hotel at all despite the fact that I watched him do it and a guy from our hotel had waited for us at three buses in the morning! After hearing our story the hotel were very greatful that we had sought out their hotel anyway and we checked in.

The DMZ day tour was great fun and fascinating lead by our knowledgable guide it was a shame however that the minbus crew decided to try and scam us at the end of the day but at least it all turned out fine!

As I write this we are waiting for our bus out of Hue to Hoi An so unfortunately I am still 3 days behind however Hue was amazing, we had a great time with our new found friends and I’ve made notes so I can write about Hue later. We are hopefully meeting up with Tom and Craig for 24 hours in Hoi An and you’ll hear all about it though probably in a weeks time!