skip page navigation Oregon State University

Planet OSL

Ganeti Tutorial PDF guide

Lance Albertson - 06/12/2012

As I mentioned in my previous blog post, trying out Ganeti can be cumbersome and I went out and created a platform for testing it out using Vagrant. Now I have a PDF guide that you can use to walk through some of the basics steps of using Ganeti along with even testing a fail-over scenario. Its an updated version of a guide I wrote for OSCON last year. Give it a try and let me know what you think!

Categories: Planet OSL

Trying out Ganeti with Vagrant

Lance Albertson - 05/20/2012

Ganeti is a very powerful tool but often times people have to look for spare hardware to try it out easily. I also wanted to have a way to easily test new features of Ganeti Web Manager (GWM) and Ganeti Instance Image without requiring additional hardware. While I do have the convenience of having access to hardware at the OSU Open Source Lab to do my testing, I’d rather not depend on that always. Sometimes I like trying new and crazier things and I’d rather not break a test cluster all the time. So I decided to see if I could use Vagrant as a tool to create a Ganeti test environment on my own workstation and laptop.

This all started last year while I was preparing for my OSCON tutorial on Ganeti and was manually creating VirtualBox VMs to deploy Ganeti nodes for the tutorial. It worked well but soon after I gave the tutorial I discovered Vagrant and decided to adapt my OSCON tutorial with Vagrant. Its a bit like the movie Inception of course, but I was able to successfully get Ganeti working with Ubuntu and KVM (technically just qemu) and mostly functional VMs inside of the nodes. I was also able to quickly create a three-node cluster to test failover with GWM and many facets of the webapp.

The vagrant setup I have has two parts:

  1. Ganeti Tutorial Puppet Module
  2. Ganeti Vagrant configs

The puppet module I wrote is very basic and isn’t really intended for production use. I plan to re-factor it in the coming months into a completely modular production ready set of modules. The node boxes are currently running Ubuntu 11.10 (I’ve been having some minor issues getting 12.04 to work), and the internal VMs you can deploy are based on the CirrOS Tiny OS. I also created several branches in the vagrant-ganeti repo for testing various versions of Ganeti which has helped the GWM team implement better support for 2.5 in the upcoming release.

To get started using Ganeti with Vagrant, you can do the following:

git clone git://github.com/ramereth/vagrant-ganeti.git git submodule update --init gem install vagrant vagrant up node1 vagrant ssh node1 gnt-cluster verify

Moving forward I plan to implement the following:

  • Update tutorial documentation
  • Support for Xen and LXC
  • Support for CentOS and Debian as the node OS

Please check out the README for more instructions on how to use the Vagrant+Ganeti setup. If you have any feature requests please don’t hesitate to create an issue on the github repo.

Categories: Planet OSL

This Makes we Want an iPad

Alex Polvi - 12/24/2011
This has to be the best iPad case on the market. Best part about it? Two good friends, Pat Buckley and Craig Dalton are some of geniuses behind. 100% made in San Francisco. Great job guys, I’m glad your ideas are paying off!
Categories: Planet OSL

shipped libcloud

Alex Polvi - 12/24/2011
Woo! We shipped an implementation of libcloud, check it out over on libcloud.org. It is a pure python library for working with a variety of cloud server providers. It is still a work in progress, but would love to get others involved!
Categories: Planet OSL

Cloudkick takes “Best in Show” at Under the Radar

Alex Polvi - 12/24/2011
Last Friday, we presented at the Under the Radar conference. Things went really well, and Cloudkick ended up taking best in show (people’s choice)! Techcrunch did a nice article on our progress. Exciting times! Here is a link to the presentation if you are interested.
Categories: Planet OSL

Launching is awesome

Alex Polvi - 12/24/2011
Today has been an amazing week — and it is only half over! On Monday, we launched our product Cloudkick. Cloudkick is a system to manage cloud servers, like those provided by Amazon EC2 or Slicehost. We have been building this over the past three months as part of the y-combinator program. It was very [...]
Categories: Planet OSL

Pico-Garden!

Alex Polvi - 12/24/2011
It was like 70 degrees here today, so I decided to plant a garden. Yep, it’s along the base of the fence line. Eh, it’s the best I can do with what I have. I’ll let you know how it works out. :)
Categories: Planet OSL

Automatic failover on EC2 with heartbeat and Elastic IPs

Alex Polvi - 12/24/2011
Put together a little hack … thought I would share … it allows you to do automatic failover on EC2, using the AWS “elastic ip” feature. Not perfect, but very cool! Posted over on the AWS forums: http://developer.amazonwebservices.com/connect/thread.jspa?messageID=112816
Categories: Planet OSL

Get around EC2 filesystem limits (sort of)

Alex Polvi - 12/24/2011
Tonight I got myself pretty excited about an EC2 hack. Essentially I was able to create arbitrarily sized root filesystems, when the limit was supposed to be 10GB… or so I thought. A little background. Amazon AWS allows you to make custom machine images to boot their “elastic compute” (EC2) nodes. Essentially they are a [...]
Categories: Planet OSL

Getting cache statistics with mod_cache

Alex Polvi - 12/24/2011
The always awesome Paul Querna came up with a cool trick to log cache hits/misses. In your virtual host set the environment variable SetEnv CACHE_MISS 1, then log it using your CustomLog with %{CACHE_MISS}e. It will log as “1″ when it is a miss, or as “-” when it gets hit. How does it work? [...]
Categories: Planet OSL

Police Scanner and GNURadio

Alex Polvi - 12/24/2011
Keeping and eye on the police has never been so much fun! Recently, my friend John lent me his USRP, allowing me to play with GNURadio. From wikipedia: The Universal Software Radio Peripheral (USRP) is a high-speed USB-based board for making software radios. GNURadio is a free software, software radio. Essentially, the USRP allows you [...]
Categories: Planet OSL

Hungry Scientist in the NYT

Alex Polvi - 12/24/2011
Check it out! Pat’s book was on the front page of the dining section in the New York Times today! I went down to Borders to get a copy, and while I was there did a little clean up of the “Reference” section… Before: After:
Categories: Planet OSL

Open Government Communities Survey; Be Counted

Deborah Bryant - 09/14/2011

If you’re involved in Open Government, I encourage you to participate in this first informal open government communities survey. The objective of the short survey is to create a view of the broad community of constituents that comprise the open government movement, with a special interest in understanding the interplay and influence of open source software and the open source community in forwarding their objectives.

The first set of responses collected by September 18th will travel to Northern Ireland for my presentation at the OpenGov Conference in Belfast on September 22nd, 2011. Results of the survey will be shared this fall on the Government Open Source Conference web site (goscon.org). Any questions? email me.

Thanks in advance or your participation!

addthis_url = 'http%3A%2F%2Fwww.bryantsblog.com%2F%3Fp%3D1318'; addthis_title = 'Open+Government+Communities+Survey%3B+Be+Counted'; addthis_pub = '';
Categories: Planet OSL

GOSCON 2011 Shakes Up Government IT

Deborah Bryant - 09/13/2011

This year’s theme for the Innovation Nation Forum, which was co-located with GOSCON, was “Shake It Up,” and things were shaken up indeed. While we were a little sad that this year’s conference wrapped up early due the East Coast earthquake, we’re all pleased that no one was harmed. Plus, we did get to shake up all things government IT and open source before the lunch hour, presenting two panels on “Cost Take Out” and “Building Outside the Box.”

You can read more from the panels in the GOSCON and GOSCON DC Tweet streams. My colleague Gunnar Hellekson from RedHat has penned an excellent post on opensource.com summarizing the Cost Take Out panel. Alex Howard from O’Relly Media moderated that panel plus did a great writeup of the event entitled “Government IT’s Quiet Open Source Evolution”.

The good folks at MeriTalk, producers of Innovation Nation, have shared a video of the luncheon exec panel which was underway when the quake hit.  More materials from the conference are forthcoming; promise they’ll be exciting in less seismic ways.

addthis_url = 'http%3A%2F%2Fwww.bryantsblog.com%2F%3Fp%3D1308'; addthis_title = 'GOSCON+2011+Shakes+Up+Government+IT'; addthis_pub = '';
Categories: Planet OSL

Government Open Source Conference D.C. Program set for August 23

Deborah Bryant - 08/09/2011

GOSCON_DC2 SqPlease join me on August 23 at the Washington Convention Center for our Seventh Annual GOSCON, this year collocated with Innovation Nation Forum.  I’m very excited we’re partnering with MeriTalk for the event which takes this year’s burning government IT issues head-on.

Our program this month includes a great lineup of all agency leadership – from the FCC to the White House, the new Consumer Financial Protection Bureau, NASA, DHS and more.  A full program schedule, registration  and other details can be found at Innovation Nation, or you can check out the GOSCON site for speaker details.  Registration is complementary to government, one registration provides access to all Innovation Nation keynotes and sessions.

GOSCON Track details:
Cost Take Out: Where are the Savings in Open Source?

  • Greg Elin, Chief Data Officer, Federal Communications Commission
  • Tiffany Smith Licciardi, eDiplomacy, U.S. Department of State
  • Dr. David A. Wheeler, Research Staff Member, Institute for Defense Analyses
  • Alexander B. Howard, Gov 2.0 Correspondent, O’Reilly Media [Moderator]

Building Outside the Box: Leading Federal Agency Innovators

  • Matthew Burton, Office of the CIO, Consumer Financial Protection Bureau
  • Scott Goodwin, Chief Information Officer for Space Operations, National Aeronautics and Space Administration
  • Dr. Douglas Maughan, Director, Cyber Security Division, Department of Homeland Security Science & Technology Directorate
  • Wayne Moses Burke, Executive Director, Open Forum Foundation [Moderator]

Open Source Lessons Learned: What the Feds can Learn from State and Local Gov

  • Carolyn Lawson, Chief Information Officer, Oregon Health Authority
  • Bryan Sivak, Chief Innovation Office, State of Maryland
  • Chris Vein, Executive Officer of the President
  • Deborah Bryant, Public Sector Communities Manager, Oregon State University Open Source Lab [moderator]

About GOSCON: The Government Open Source Conference (GOSCON) program is produced by Oregon State University Open Source Lab as part of its mission to educate and build community. Since 2005, GOSCON has helped fuel the adoption of open source technology in the public sector by attracting information technology leaders worldwide to its annual event. Ongoing conference content includes lessons learned in the development and integration of open source solutions into agency environments, exposure to projects and existing software applications and services, and opportunities to establish and foster relationships for collaboration around shared interests.

addthis_url = 'http%3A%2F%2Fwww.bryantsblog.com%2F%3Fp%3D1280'; addthis_title = 'Government+Open+Source+Conference+D.C.+Program+set+for+August+23'; addthis_pub = '';
Categories: Planet OSL

Ganeti Web Manager 0.7

Peter Krenesky - 06/18/2011

Ganeti Web ManagerWe’ve just release version 0.7 of Ganeti Web Manager. Ganeti Web Manager is a Django based web application that allows administrators and clients access to their ganeti clusters. It includes a permissions and quota system that allows administrators to grant access to both clusters and virtual machines. It also includes user groups for structuring access to organizations.

This is the fourth release of Ganeti Web Manager and it contains numerous new features.  It also includes various bug fixes and speed optimizations.  Here is the full CHANGELOG, or read on for the highlights.

Xen Support

Ganeti Web Manager now have full Xen support.  Prior versions could display Xen instances, but now you can create and edit them too.  This as an important addition because Xen is a widely used and mature project.  Now with full hardware virtualization in Linux 3.0, Xen will continue to be an important technology for virtualization.  This was our most often requested feature and we’re glad to have fulfilled it.

Internationalization

Thanks to a large community contribution, internationalization support was added for nearly all aspects of the interface.  Users can switch between their default language and any other.  Currently only a Greek translation is available, but we’d like to see many more languages. If you can read and write another language this is a great opportunity for you to get involved. We’re using Transifex to coordinate people who want to help translate.

Search & Improved Navigation

Administrators of larger cluster can now find objects easier with our search interface.  It includes an Ajax auto-complete feature, along with detailed results.

We’ve also added contextual links wherever we could.  This included ensuring breadcrumbs were properly formatted on each page.  Object Permissions and Object Log were updated to ensure navigating between those screens and Ganeti Web Manager is seamless.

Import Tools

There are now import tools for Nodes.  These work the same as for instances.  The cache updater has also been reworked to support both Nodes and Instances.  It’s now a twisted plugin with modest speed improvements due to Ganeti requests happening asynchronously.

Speed, Scalability, and Bugs

We’ve sought out places where we performed extra and or inefficient database queries.  We identified numerous places where database interaction could be reduced, and pages returned faster.  This is an ongoing process.  We’ll continue to optimize and improve the responsiveness as we find areas of the project we can improve.

Numerous bugs were fixed in both the user interface and the backend.  Notably, the instance creation interface has had several bugs corrected.

Module Releases

We’re building several modules along with Ganeti Web Manager.  The following projects have new releases coinciding with Ganeti Web Manager 0.7:

Django Object Permissions 1.4

  • improved user selection widget
  • speed improvements

Object Log 0.6

  • our first public release
  • speed, scalability, and flexibility improvements

Twisted VNC Auth Proxy

  • our first public release
  • added support for hixie 07 and latest noVNC version.
Want to learn more?

Lance Albertson and I will be speaking about Ganeti & Ganeti Web Manager at several conferences this summer.  Catch us at the following events:

Categories: Planet OSL

Google I/O 2011

Peter Krenesky - 05/18/2011

Google I/O LogoFive OSUOSL co-workers and I recently finished a road trip to Google I/O 2011.  We took two cars on an 11 hour drive through scenic southern Oregon and northern California.  We learned more about Android and other technologies shaping the web.  It was also a great opportunity to spend time with each other outside the office.

Monday night we joined about 30 Google Summer of Code mentors for dinner and drinks hosted by the Google Open Source Programs Office.  We’re always grateful for events that bring together friends old and new.  One developer nervously sat down at our table, professing that he didn’t know anyone.  We might not work on the same project, but we’re all part of the open source community.

The highlight of the conference was the double announcement of Android Open Accessory program and Android @ Home.  Both open up Android to integration with third party devices.  These features coupled with near field communications (NFC) stand to dramatically change how we use our mobiles devices to interact with the world around us.  This is not a new idea.  X10 home automation has existed since 1975.  Zigbee and Z-wave are more modern protocols, but also available for years.  The difference here is 100 million Android users and a half million Arduino hackers.

As Phillip Torrone wrote on the Makezine Blog, “There really isn’t an easier way to get analog sensor data or control a motor easier and faster than with an Arduino — and that’s a biggie, especially if you’re a phone and want to do this.”

It won’t be a short road.  We still have obstacles such as higher costs.  A representative from Lighting Science I spoke to at their I/O booth quoted Android@Home enabled LED lights at $30 per bulb.  Android and Arduino might be the right combination of market penetration, eager hackers, and solid platforms for a more integrated environment.

NFC Sticker

My favorite session was How To NFC.   NFC (near field communication) is similar to RFID except it only works within a few centimeters.  Newer android phones can send and receive NFC messages any time except when the phone is sleeping.  NFC chips can also be embedded in paper, like the stickers that came in our I/O Badges.  An NFC enabled app can share data such as a url, or launch a multiplayer game with your friend.  It makes complex tasks as simple as “touch the phone here”.  Android is even smart enough to launch an app required for an NFC message, or send you to the market to install the app you need.  Only the Nexus-S supports NFC now, but this feature is so compelling that others will support it soon too.

The other technical sessions were very useful too, whether you were interested in Android, Chrome, or other Google technologies.  The speakers were knowledgeable on the subject areas they spoke on.  I attended mostly Android talks, and it was great hearing from the people who wrote the APIs we’re trying to use.  The sessions were all filmed and are worth watching online.

Categories: Planet OSL

PyCon2011: Snakes in a [Mother$#@!in] Brain

Peter Krenesky - 03/16/2011

I just returned from PyCon 2011, the largest annual gathering of Python users and contributors.  The conference was full of energy and I came home with my head stuffed full of new ideas and Python skills.  Hillary Mason best described my feelings about PyCon in her opening keynote, “I’m glad I’m in a room where list comprehensions receive spontaneous applause”.

The Talks

The talks came in many flavors: hands-on tutorials, sessions, a poster session, and open space discussions.  Topics included dev-ops, deployment, scalability, concurrency, large scale data processing, science, and much much more.  There was a great deal to learn for both novice and experienced programmers alike.  Most sessions taught useful skills like:

But some sessions were just fun, mind blowing examples of what you could do with Python:

It was difficult to choose which talks to see during most time-slots.  There were just too many great topics to choose from, so it’s fortunate that the session videos are already online.  Many thanks to the PyCon team for being so prompt.

The Jobs

It’s an exciting time for Python developers whether you are just entering the workforce, or looking for something new and exciting.  Part of the exhibit hall was dedicated to startups looking for new employees, but every other exhibitor was looking for employees, too.  There is definitely an employer out there to match your individual passions, and I’m glad to know that my students will have many great choices after graduation.

The Hallway Track

This was my first PyCon and did not know many other attendees, so I planned to spend a good deal of time meeting other people.  Among the 1380 attendees walking the halls and attending sessions were Python Core Developers, authors of your favorite libraries, keynote speakers, and even Guido.  While this could seem intimidating, we all came to PyCon to learn from each other and collaborate. Everyone was welcoming and happy to share knowledge and great conversations.

Six days of talking to random people resulted in many awesome “a ha” moments.  Topics spanned programming, technology, science, and art.  The ideas I shared in these talks were as valuable as the formal sessions I attended.  The best part was making so many new friends.  Looking forward to a great PyCon with you all next year in Santa Clara!

Categories: Planet OSL

Ganeti Web Manager 0.6

Peter Krenesky - 03/04/2011

We’ve released Ganeti Web Manager 0.6.  Ganeti Web Manager is a Django based web application that allows administrators and clients access to their ganeti clusters. It includes a permissions and quota system that allows administrators to grant access to both clusters and virtual machines. It also includes user groups for structuring access to organizations.

This release comes after a short development cycle, with the goal of fixing critical bugs and providing important core features.  Check out the full change log, or read on for some highlights:

Virtual Machines

Ganeti Web Manager 0.6 includes multiple improvements to the virtual machine detail view.  We’ve added the complete list of virtual machine properties.  The layout has been updated to group properties into relevant sections, as well as make it more readable.

The following new controls were added for virtual machines:

  • Edit a virtual machine’s settings.
  • Rename a virtual machine.
  • Migrate a virtual machine to it’s secondary storage.

Ganeti Web Manager 0.6 also features improvements to the virtual machine deployment process.  It now detects and recovers from Ganeti errors better than before.  If a create job fails without ganeti deploying the virtual machine, you can edit the settings and re-submit the job.  All other failures will let you continue to the virtual machine detail view where you can use the provided admin tools to repair the virtual machine.

Nodes

node detail view

Nodes are now cached by Ganeti Web Manager.  This allows views using node data to be displayed faster. We now also provide Node views that allow an admin to issue commands on a node such as migration and changing the node role.  The node detail view also provides information from the perspective of a node including used resources and which virtual machines are deployed on it.

Logs

Ganeti Web Manager 0.6 now provides a log of actions performed on every object.  This will allow admins to see the history of every action taken on a VirtualMachine, Node, and Clusters.  It also shows every action a user account has performed.  The log is intended to aid auditing and troubleshooting.

Logging is provided by the newly branded Django Object Log app.  It is a reusable app that can log generic messages.  Each message can define it’s own rich format, including contextual links to the related objects.  Object Log will be developed in parallel with Ganeti Web Manager and future projects by the OSUOSL.

Categories: Planet OSL

vcl_hash

Greg Lund-Chaix - 10/14/2009

I’ve been trying to figure out the effect of this line in my Varnish config:

sub vcl_hash { if (req.http.Cookie) { set req.hash += req.http.Cookie; } }

It seemed to make sense, but I was having a hard time wrapping my head around its ramifications. I was looking at some of the docs on the Varnish site and at this great Varnish config walkthrough when the metaphorical lightbulb went on. By adding the cookie to the hash it’s effectively creating a per-session cache.

Hmm. An interesting tradeoff. On one hand it’s filling up my available cache with duplicate copies of the same content because the hash identifying the cached content is cookie-specific. On the other, it is delivering content from cache that wouldn’t normally be cached because of the cookie.

Share this: Digg del.icio.us Facebook Google Furl Print this article! Reddit Slashdot StumbleUpon Technorati TwitThis Fark LinkedIn Ma.gnolia NewsVine Ping.fm Pownce Tumblr

Categories: Planet OSL