It has been awhile and a lot of things have been done in my cloud. My goal with this post is to first provide an update regarding the current state. Provide some lessons learned from an overall view. Second I want to redefine and reevaluate the direction of my cloud/IT in a box. Finally, I have some, what I think is some exciting news, potential growth opportunity, and great way to follow the work I’m doing. Maybe even a way for you guys to participate and help learn and shape this private cloud.
Current State
OpenStack
OpenStack is doing really well in my setup. It is the heartbeat of my infrastructure. It is highly customizable, flexible, easy to use, but boy does it have a bunch of moving pieces. And I understand why it is that way. The OpenStack project is trying to deliver a flexible, choose your own adventure style private cloud. By no means a small feat. Without going into detail, here is my quick synopsis. When it is working, it is an awesome suite of tools that allows so much power at your finger tips, with a great supporting community. When something is acting up or isn’t working, it is painful to debug and the docs aren’t exactly easy reads. Despite that, I still highly recommend OpenStack over VMware in terms of VM based cloud computing setups.
FreeIPA
FreeIPA (Identity, Policy, Audit) is a really great all in one tool for SSO, DNS, cert management, and policies. I’m sure there are features I’m not even leveraging, but all in all, it is a great tool that is making the collaboration of various standards and other security based tools much easier to implement and use. I have deployed VMs register to FreeIPA so I can bypass the ssh key requirement of OpenStack, especially if I misplace the pem file itself. Additionally it is giving me hands on experience with domain and ldap management and forcing me to think about security and policies in terms of ease of management and use while still being secure. Not an easy puzzle. But I feel like I have the ability to treat my environment like an enterprise, without the price tag that usually goes with it.
Hashicorp Consul
I am admittedly not using this as much as I should. But I’m hoping to change that soon. Consul is a highly available Key Value and Service Discovery platform. I’m currently using it as a backend for Hashicorp Vault and Terraform, and light service discovery and service monitoring. I have plans on leveraging this much more, which I will elaborate more on when we look forward.
Hashicorp Vault
I have begun ramping up using this more in Terraform to start. My goal is to leverage this much more for new stuff, and convert older things to start using it as well. My initial goal is to keep all secrets out of code and in the Vault. Eventually I will leverage it with rotating tokens, and for container deploys in Kubernetes.
phpIPAM
A great tool for tracking what you have on the network. Sadly though, another one that went to the wayside during heavy development. I have a script to have VMs register themselves to it, but I tend to opt out of dealing with it when testing deploys. I am currently looking into a way to have this registration happen on OpenStack deploys no matter what, I’ve posted a question to ask.openstack to see what is available: https://ask.openstack.org/en/question/113434/have-openstack-run-script-on-every-deployprovision/
iTop
Another tool I’ve been neglecting, but I don’t consider any less useful. iTop is a free CMDB tool. I’m a huge believer of making sure you know what you are affecting when you do things. The CMDB is the place to track those relationships. But it is not an easy thing to dynamically figure out and get tracked. If you were to sit down and think about every single thing and what they each talk to, especially in an enterprise-like environment, it would blot out the sun. Especially considering base services that service or the potential to service everything. FreeIPA is a good one, as it touches every single machine for not only SSO, but CAs and DNS as well. Combine that with OpenStack in my environment (no getting around that), but even down the hosts themselves. Service the OS of one of the OpenStack nodes, or network switches themselves and affect so many services. All the more reason to track it, as there will always be that one thing you forgot was connected to what you work on. But like phpIPAM, i turned off the registration.
OpenVPN
Nothing really to note on this guy. He sits there providing VPN services. I did get him hooked into FreeIPA so I can login with LDAP credentials from outside, in addition to the certificate checks. Great piece of software for free.
Gerrit/Git
I love Git. While Git does have a steepish learning curve, they do things a certain way for a reason while allowing flexibility. Even though the coding review capabilities of Gerrit is really only used for myself, but now by one other person as well. It allows visibility and the chance to ask question, helping to supplement the learning opportunities that would otherwise make you have go out of the way to create.
Kubernetes
This is still relatively new but opportunities are for making CICD processes easier and painless are all around. My only complaint is some of the external networking is a little cumbersome. I would like to see the ability for DNS to be updated by a service deployment. Such as when I deploy Phabricator, I would have liked to have the ability to have K8s update an external DNS so I could go to phabricator.myFQDN much like Google Cloud allows. Maybe there is a way and I’m not thinking of it, or just not aware. If you know, please comment below.
Phabricator
Another thing relatively new and still figuring out the best way to use certain parts. Phabricator though is a great all in one tool that is free to use, no limitations. I’m honestly surprised I’m not see more things about it. It has a Wiki, Task Tracker, Blog interface, Q&A sections for groups, even a code repo, code review, and code audit. (FYI: I chose not to use the code portions of Phabricator because I did not like the required use of their client-side tool Arcanist. While there is git-review that can be used for Gerrit, it is by no means required.) Also MediaWiki has some great articles written on how they are using Phabricator. Even better than some of Phabricators own documents.
Looking Forward
So what’s next? I would like to list things in order for where I plan to take this setup. First, what is the destination? Why am I doing this, and when is this considered done and in a maintainable state? What is my Mission Statement.
My Mission Statement: To have an Enterprise-like development environment for web & application development, machine-learning, and data science.
With that in mind, what do I need to do & setup and why to get to that end goal? What does that end goal look like from a tech perspective?
Tech Perspective Goal:
I like the idea that OpenShift.io is putting together. An integrated dev environment that lives in containers, and hooked into Continuous Integration and Continuous Deployment tool sets for better enabling of those practices with DevOps. Below is the list of technologies I would like to implement to get to that point. The Development section is the final top layer, while the other sections below it, I feel are worth setting up to help enable it. The desire is that once this is stood up, to start on some coding projects I have had ideas for.
Development
- Fabric8 – Integrated Development Platform
- Eclipse Che – Web Based IDE
- OpenShift Origin – Container Application Platform
- Jenkins or Concourse CI – Automation Server
- Spinnaker – Continuous Delivery
- SonarQube – Continuous Code Quality
- Coverage.py – Code coverage testing for Python, integrated with Automation Server
Monitoring
- Elastic Stack – Log Analysis
- Prometheus – Service monitoring and metrics
- Grafana – Metrics Dashboard
Configuration Management – Need to decide what I am using for configuration enforcement and not just initial deploy
Need to Spike and requesting suggestions:
Free binary repository for all matter of binaries. Docker containers, rpms, debs, self made bins, etc. Preferably similar to Artifactory where if it is hosting RPMs, then it is presenting a Yum Repository, etc.
Exciting News
I’m proud to announce that I will begin Live Streaming via Twitch the setup of the tools up above. On Sunday I will be Live Streaming for a couple of hours as I work to implement the technologies above. At this moment no schedule has been defined, yet! Once I have settled on a time, I will announce it via LinkedIn and Twitter. I will also be looking to setup a Patreon if you would like to support this and have some influence on the order and development of features for this stack. Further details will be announced once that have been made official. I’m hoping this will be a learning opportunity not only for myself, but for you as well. I want to leverage the platform to inspire and cultivate technical knowledge and I hope to produce something that you would like to join in on. Feedback is always welcome, and I hope you are able to join me this Sunday!