It has been awhile and a lot of things have been done in my cloud. My goal with this post is to first provide an update regarding the current state. Provide some lessons learned from an overall view. Second I want to redefine and reevaluate the direction of my cloud/IT in a box. Finally, I have some, what I think is some exciting news, potential growth opportunity, and great way to follow the work I’m doing. Maybe even a way for you guys to participate and help learn and shape this private cloud. Continue reading Private Cloud update and reconciliation
Category Archives: Blog
Cisco Small Business SG 200-08 Switch Shenanigans
First, the problem. As I have indicated in my previous article (Private Cloud Architecture) I have 4 Cisco Small Business SG 200-08 Switches that help make the network backbone of my Private Cloud. One issue I have had from day 1 of this setup is the inability to use NFS or RPC based technology. After getting frustrated at both lack of available verbosity surrounding why NFS was timing out, as well as my lack of knowledge surrounding NFS and RPC technology in general (I know how to do a simple NFS mount server and client, but nothing below that in the technology stack), I shelved the initiative to use NFS as well as the shared storage problem between Nova Compute Nodes. Fast Forward almost a year, I was finally at the point in my infrastructure that I was going to need to try it again. I tried many things and was given the suggestion, verify what you know in a known environment and slowly migrate it into the problem environment. Continue reading Cisco Small Business SG 200-08 Switch Shenanigans
A setback and the importance of backups
One would think, that someone who pushes the importance of backups so hard, regardless of HA and DR (which I maintain to this day), that they would have backups in place for their own infrastructure, right? Well, I didn’t, it was on my docket, but like many before me in their own respective environments, I pushed it off for a myriad of reasons. A couple of those revolve around the fact that I can not for the life of me figure out why whenever I attempt to mount NFS hosted from my FreeNAS server to any other box, it times out with the only error and log message being that it timed out with no lead in. Another reason being that I was working on stuff that was far more exciting, which is a low bar to begin with. Anything is more exciting than backups. And because of my rush for getting to the seemingly greener grass on the other side of the fence, I lost almost everything. My only saving grace is the inherit nature of Git being decentralized.
Alright, so enough beating myself up for my stupidity, let’s go over what happened, what I did, and what I am going to do. My OpenStack cluster was built using Packstack and the RDO project. Which for me, was a great way stand up OpenStack quickly to learn the ins and outs of OpenStack and get familiar with it without getting overwhelmed from the complexity that is OpenStack. I was looking to deploy Kubernetes within OpenStack in order start working with containers. However, the Heat service was not installed, which was required to deploy Kubernetes quickly. So I edited my Packstack answer file to enable HEAT and reran Packstack. It failed because Keystone was throwing errors. Mainly it couldn’t find a column in a table of its database. This only started showing after I ran Packstack. So I started tweaking by hand and came across instructions for v3 of the keystone API, but not v2 which is what Packstack had installed. So I attempted to upgrade Keystone to v3. Upgraded fine, but it was still throwing the same database error.
After a few hours of fighting, and coming up with no reason for why the database would be missing columns, I gave up. I didn’t know enough about the database to be able to fix the problem, and my database/SQL-fu is lacking. I quickly became depressed shortly after making the realization that without Keystone working, absolutely nothing in OpenStack worked. Neutron, Nova, Horizon, everything depends on Keystone. Which judging by its very name I should have been able to guess even with no OpenStack experience. After pondering for a bit, an idea occurred to me. Continue reading A setback and the importance of backups
Apologies for slow updates
A lot of things have been happening. First, one of my builds (my storage server) continued to have a faulty motherboard that I had to keep exchanging and continued to be DOA. What made this motherboard more tempting was that the motherboard was mini-itx, had more than enough SATA ports on the board making it so I didn’t need an additional hard drive controller card, integrated CPU and fan less heat sink, ECC memory support, and finally IPMI. However because I was always getting DOA boards I contacted NewEgg and despite the board only having a replacement only warranty they allowed me to instead turn it back in for store credit so I can instead purchase a different motherboard. The new parts arrived and I will be putting it together tonight and will be taking notes to publish my build guide.
All while I was dealing with a bad motherboard I ran into a problem with my OpenStack dev box. The block storage service known as Cinder was not properly configuring persistent data drives that I wanted to use. They were always created with an error. With that came a revelation to me that I was not properly learning OpenStack and its commands because I had let PackStack configure it for me. While easy and relatively quick, left me feeling unsure if I was doing things right in my setup and even more confused and unsure when something was broken. In response to that I decided to pick up a course for OpenStack through Linux Academy. So everyday I’ve been watching videos on the bus ride to/from work and doing the excercises when I got home. I’ve been learning a lot and plan to put it all to use when I create my production instance of OpenStack home cloud.
With that said I am getting close to finishing the next entry in my Private cloud series, which will be configuring the network switches and the hosts. I’m looking forward to posting what I learn and hopefully helping you make your own as well.