My First OpenStack Summit: Tokyo 2015

In October I attended my first OpenStack Summit. It's taken me awhile to digest and reflect on my experiences hence this article being published over a month after I got back!

I'm not new to tech conferences or contributing to OpenSource, so much of the "Summit 101" stuff didn't really apply to me. That being said, it's worth noting that the OpenStack Summit has essentially 2 tracks, one for users that's more marketing focused, and one for contributors that's more developer or operations focused. I spent pretty much all of my time in the Developer Summit, so I can't speak to the other sessions.

I think the most important thing to remember when attending something like an OpenStack Summit is you need to have something to focus on if you want to get anything out of it. Sure you can float around and fly by the seat of your pants, but at least I'm the type of person that finds it hard to really ingest things unless I have some kind of context or goal to fit it into.

As a developer, I was hired by my current employer to work on OpenStack Infrastructure upstream projects. After some limited involvement with the Infrastructure team, I started getting involved with Nova (aka Compute). This is largely because I'm more of a traditional Software Engineer and generally found the Nova team to be a better fit for my work style and interests. I feel pretty lucky to be in a work setting where making this kind of transition is so easy compared to other places I've worked! So I guess the first piece of advice I have, if you're hired to work on anything, whether it's upstream on an OpenSource project or on internal proprietary things, if something else catches your interest and seems more relevant to your interests than whatever it is you're currently working on, then definitely look for opportunities to change your focus!

Going into the Design Summit, I had just started working on Nova about 2 weeks prior. My goal for the Summit was to gain as much context as possible, and to meet with my new team members face to face.

What I Learned

Reviews are Important

Even if your goal isn't to become a core member on a project, participating in the review process is really vital to keeping up with what's going on in the project. When you review someone's patch, you have to look at code you might not normally look at and learn how it works in order to provide useful feedback. Reviews help you to understand the code base and to participate in the project. In the long term, if you regularly do useful reviews, you might even be considered for a core role on the project (which could be good or bad depending on how much time you have to spend on the project haha)! I'll write an article that's more in depth on doing reviews.

Development Happens in Cycles

Development typically happens in 6 month cycles with milestones identified by the PTL or other core members. This can include a cut off date for proposed specifications and cut off dates for releases. Additionally sometimes there are special sprints for things like bugs and documentation. Teams typically announce this on the mailing list and may link to an etherpad or two with this information.

Be the Change

The number one thing I heard repeated during these sessions was "propose a spec and we'll discuss it". If you have an issue and you want it to be a priority, the number one way to get the most out of your face time with the development team is to propose a specification for it. Don't plan on having a presentation with slides or a demonstration as media equipment and switching laptops could be tricky. Have an etherpad ready and have links to a blueprint in launchpad and a spec in review.openstack.org. Even if they are in a draft state, it's an artifact people can look at while you're talking about your thing. Finally, expect to do the work for your change. If you can't do it, then do your best to stir up interest before you present the idea.

Where to Find a Team's Priorities

Priorities are the most important items that need to be completed in the next cycle. The goal of the developer sessions in the summit is to discuss issues that are being proposed and determine if they are a priority. If they are not deemed a priority, they can still make it into a release if they have sponsorship, but priorities are going to get the most attention from team members. The team typically provides a link to their priorities list etherpad on the mailing list but you can also find it during sessions. During the last day of the summit, this priority list becomes the agenda as items discussed during the week are revisited.

Non-priority Items are Important Too

As mentioned before, just because something isn't a priority, it can still get attention and make it into a release. If you are willing to champion the issue and get the work done, the only challenge will be getting core attention to push it through to release. In the worst case scenario, you get all the work done and you don't get the attention you need to get it approved and landed. In that case, you just have to wait until the next summit and pitch it again, reminding everyone that the work is done. The summit will also be a good place to get some face time with core members that can provide feedback that might be more challenging to get remotely with barriers like time zones.

What Could Be Improved

I think the biggest thing that could be improved about the developer summit format is the way communication is facilitated. I was very fortunate and spent most of my sessions with a team that has a PTL who is clearly experienced as a facilitator. Unfortunately that is not true for many of the teams and it creates a very frustrating experience for people whose communication styles do not match an extroverted "high involvement" type but follow a more introverted "high politeness" type [1].

Basically the PTL pulls up an etherpad with a list of items to discuss. People in the room talk when they want to. Some people tried raising their hands to signal they had something to say and in some cases when the discussion got very involved, were passed over. Or worse, when they did get the chance to speak, they were cut off before they could finish. While this happened less frequently in some sessions than in others, it happened more frequently than it should have. Many valuable insights and opinions were basically lost.

The most difficult situations happened when people didn't agree on something. I saw some rude behavior that I found rather surprising. Many of us are working for companies who are paying us to be involved in these projects and we are, however indirectly, representing that company. Making backhanded comebacks might make you feel good about yourself in the moment but ultimately it kills discussion and prevents people from listening. I saw this mainly with cross-team discussions where clearly there was some history, but I did see it in at least one team discussion. It's less important for a person to be *right* than it is for the group to come to a good decision about a thing. At the end of the day no one remembers if someone was right or wrong, we remember what was ultimately decided and go on with our business. People do, however, remember if you were a jerk.

I think some kind of group communication 101 and communication guidelines would be really valuable for both participants and PTL's. Additionally, facilitation training for PTL's and other moderators would help to improve the overall discussion flow a lot. This still surprises me in many industries but in tech especially how we not only tolerate antisocial behavior, but practically celebrate it. A person can be smart and be making valuable contributions but if they are driving away contributors, that's a net loss. Engineers tend to be focused on developing technical skills, not social ones. There are a lot of social basics that we don't seem to possess and this is a problem across the industry, not just in OpenStack.

What Was Good

Everyone I met on the Nova team was super nice and fantastic. I felt as a team they were very welcoming and their sessions were some of the best facilitated that I attended. I actually shared my anxiety about participating in discussions with the PTL. He was very nice about it, admitting he'd felt the same way when he started and was overall very encouraging. Another core contributor I've been working with made a point to call me out when something came up that I'd started helping him with, which also helped to break the ice for me, making it easier for me to introduce myself to others in between sessions.

I got a ton of context from the Design Summit sessions. I left feeling so much more in touch and in tune with what the community is doing that now when I'm in irc or reading in the mailing list, I have a better idea of what a person is referring to. As a new contributor, I found a couple of things to get involved with and left the summit with a ton of enthusiasm and energy to dive in.

For me the best thing about the Summit was getting to meet the people, in person, on the team I recently started working. I got to have lunch with a few of them on the last day and I really enjoyed myself.

In general, everyone was so friendly and I'm really looking forward to the Nova Mid cycle.

Final Thoughts

I know I ranted a lot about some of the downsides of the Summit, but to be fair, this is not at all unique to the OpenStack community. I don't want to discourage anyone who is thinking about participating, but I don't want people to believe any nonsense from anyone trying to say that OpenStack somehow doesn't have these issues. Everyone has these issues, including every company I've ever worked for. Personally I think the road to improvement is a focus on general inclusiveness rather than targeted demographics. I've provided some ideas on how communication could be improved as a step in that direction and I look forward to seeing how things evolve.

It's truly amazing that companies are seeing the value of Open Source enough to pay people to work on it full time. I'm also amazed that so many employees of so many different companies, representing so many different interests (and nationalities) can come together and work together to build a pretty neat thing that so many people are using! That's a real accomplishment and I'm definitely looking forward to my next Summit.

[1] The labels "High Involvement" and "High Politeness" (aka "High Considerateness") originated with the Sociolinguist Deborah Tannen. You can read a book of her theories, Conversational Style, or here's the short version on Wikipedia.

My First OpenStack Infra Patch

In August, I started a new job as an upstream contributor on the OpenStack Infrastructure team! The project is so huge, the average on-boarding time is at least 6 months. My goal for this on-boarding time is to document as much as possible while I still have my "new contributor" perspective. I have been focusing on documentation for various Infra projects, including Nodepool and Disk Image Builder.

I had my first OpenStack Infrastructure patch merged on September 2nd, 2015! I'd like to share my experiences writing, submitting, and debugging my first patch.

Step 1: Figure out what to patch

There's a talk I give about how to get started with contributing to the Perl community. Some of the advice I've given there holds true to pretty much any project, but especially any project that is big and overwhelming. Sure, you can submit code as your first patch. For most of us, however, there's a lot of context missing when we interact with a new code-base (and Infra has MANY contexts and code bases). I recommend what I like to call the "Wax-On Wax-Off Paint-the-Fence" approach.

Take on the most ridiculously boring and tedious tasks while you learn how things work.

Good examples are things like adding or improving documentation, adding, fixing, or otherwise improving unit tests, or small code tasks like variable or function renames, formatting changes, whitespace cleanup, or logging or other text output format changes. Don't do anything too invasive until you understand how all the pieces fit together. You can't debug a failure if you don't know where to look, and this is especially true for big projects like Infra!

The team I joined has a policy of assigning an onboarding buddy to new hires. I was lucky to get to work with Greg Haynes. He's done a great job of suggesting tasks like the ones mentioned above that exist in the code bases he's most active in.

My first patch involved reformatting text in a README file for one of the elements in Disk Image Builder. A proposal was made to move this from free text that was all over the place and inconsistent to a formatted table structure with clear items that should be listed for each element. This is a very tedious project and not particularly interesting to someone who is very familiar with the code base. For someone like me, however, it's perfect! So that's my current project, reformatting and improving the documentation for the environment variable overrides in Disk Image Builder's elements.

If you don't have a buddy to make recommendations, just pop into the #openstack-infra channel in IRC and just ask for suggestions! Alternatively, try the #openstack-doc and #openstack-qa channels.

Step 2: Set Up Gerrit and git-review

OpenStack has done an excellent job of documenting the steps for new developers and I recommend following those instructions after reading some of my notes here. There are more in-depth instructions provided by the Documentation team for first time contributors that may be a better fit if you don't have a lot of experience contributing to projects!

OpenStack uses a code review tool called Gerrit hosted at review.openstack.org and uses Ubuntu Single Sign On to manage login credentials. When I joined the Infrastructure team, I already had an Ubuntu Single Sign On account via Launchpad from contributions I'd made to Ubuntu a few years ago.

Additionally, to contribute to any OpenStack project (including Infra), you'll need to create a community account and sign the agreement. Make sure the email you provide for your OpenStack email address matches your Ubuntu Single Sign On email address! I ran into this issue myself and I've seen it enough that I'll bet it's one of the top new contributor issues! If you have any issues with your OpenStack community account, the first thing to check is that those email addresses match.

Finally, install git-review on your system. As a side note, for development projects, I generally try to minimize system-wide installs that could affect my entire system. This is especially important when dealing with several projects that may have conflicting version requirements. That being said, git-review is fine to install system-wide, just be aware in the future when some instructions tell you to install something system-wide and think about possible conflicts with any other projects you might be using. See the next section for suggestions on not littering dependencies all over your local file system.

Step 3: Get the Code

Infra code, like all OpenStack code, lives in a git repository hosted on git.openstack.org. To find the repository for the project you want to patch, look up the project by name at git.openstack.org using the Search box at the upper right corner. Copy the “git://” URI to clone the git repo locally.

For your local dev setup on your local machine, I recommend creating a folder for all of your OpenStack related projects. I have a folder called "dev" in my home directory that contains subfolders for various projects. Under that, I created a folder called openstack where I clone my git repos. For OpenStack infra, you will want to create a separate openstack-infra directory because openstack-infra can have different dependencies than other OpenStack projects. It should use its own virtualenv to manage these dependencies.

In general you should be using a local Python rather than a system Python. You know if you're using system Python if you have to type "sudo" anytime you want to do anything beyond running a program (like pip). Virtualenv makes doing this really easy. Learn more about using virtualenv here.

There's another great tool for managing environment variables that's written in bash, called smartcd. It lets you set up custom environment variables on a per directory basis. A lot of OpenStack projects depend on environment variables so having something to customize this without littering your environment is really useful.

I also recommend that you set up a virtual machine with a Devstack. If you're editing docs or unit tests, it's not too big of a deal to run things locally but when you're doing larger code patches, you want to mimic running tools against an OpenStack as much as possible. For the documentation patches I've done, running locally and using a virtualenv has been sufficient.

Step 4: Run the Tests

Once you've cloned the git repository, follow the instructions in the README at the parent level to do any necessary local setup. It will likely involve installing a bunch of python dependencies. See the section above for my recommendation involving using virtualenv for this.

If the project you're working on does not have a README, or the README doesn't mention anything about how to set up the repo, this is definitely an opportunity to add a patch. Every project should have a README and it should contain not only setup information but also bug submission information. These are good first patches as you get to know the infrastructure.

Some projects are complex, require a lot of setup and possess a huge range of dependencies. If your change doesn’t actually require running the project, don’t get overwhelmed by getting the project up and running in order to submit your first patch. You can easily run a subset of the unit tests locally to verify your change without having to locally test the entire project.

After configuring and installing, before touching anything else, I highly recommend running the tests to make sure everything works on your system. It's incredibly hard to troubleshoot when you're not sure if your patch broke something or if it was a setup issue that's causing test failures. To run the tests you'll need to set up tox if you haven't already. It's really straightforward (pip install tox), see the OpenStack Python Developer Docs if you're not sure how.

The tox.ini file at the parent level of the repo defines the different sections that tox recognizes. So if all you really care about is docs, there should be a section called "docs" and it will be clear in the tox.ini what tests are being run as part of that. If you are only touching a specific subset of things (eg, docs), then don't worry about troubleshooting if the entire test suite fails on your local machine.

Unit tests can fail for a variety of reasons, including versions of dependent libraries and OS-level conflicts. This is why it’s important to a) use a tool like virtualenv to separate Python language dependencies and b) to use a virtual machine or other container to avoid system level conflicts that are hard to debug.

Step 5: Hack Hack Hack

Before you start hacking, you should cut a local branch to commit to. However, if you suspect an upstream conflict is going to be merged while you’re working, you can hack on master without committing, and then cut the branch once you're ready to commit. The advantage of just hacking on master (without committing) is you can easily keep the branch current without dealing with merge conflicts. If you accidentally commit to master, Gerrit won't let you merge when you check in your commit. You’ll need to cut a local branch to submit your code upstream anyway, so it's generally just good practice to work on a local branch.

Step 6: Run the Tests

After you're done hack hack hacking, run the tests again to make sure you didn't break anything. Again, only run a subset if that's all you care about. This will just catch anything specific to what you're changing. Jenkins will run the full suite of unit and integration tests once you submit your patch for review.

Step 7: Compose a Commit Message and Rebase

Composing a well thought out commit message is important, even for small changes. The typical OpenStack commit style is to do a short description followed by a longer paragraph that provides more context for the change. If the change is really minor, you can get away with a short description. Learn more about writing good commit messages here.

Gerrit will create one review item per commit. If you have multiple commits, you need to rebase them down to one commit. My workflow is to write my real commit message for the first commit and then do short, less descriptive commits for anything subsequent. Once I'm done developing, I do rebase -i HEAD~n where n is the number of commits (inclusive) to rebase. When you do an interactive rebase, you can tell git what you want to do with each of the commits.

It might be tempting to just constantly rebase your work into one commit to save yourself the trouble. While doing your initial work, I don't recommend this because you might make a mistake that's hard to back out of without your full commit history.

To keep a remote copy of your branch without having it go through testing, you basically push your change to Gerrit but then mark it as a Work in Progress (WIP) through a workflow comment. Run git review, then open the patch's page in Gerrit. Click Review on your patch and mark it as "-1 WIP". This will let everyone (including Jenkins) know that this patch isn't ready for testing just yet.

Note that if you want to keep a remote copy of your work in Gerrit, you'll need to rebase it down to one commit. I haven't figured out an alternative solution that keeps the repo in sync with the master branch but lets you have a remote copy of your revision history without creating a bunch of extra patches in Gerrit. When I figure something out I'll be sure to talk about it in a future article.

Step 8: Submit Your Patch

Before running git-review, make sure the email address in your git config matches the one you have registered in Gerrit. Just type "git review" to submit your patch. If you want to submit the patch as a work in progress, see the above section about doing a workflow commit.

Once you’ve submitted your patch for review, the build system will run a huge suite of tests on the remote branch in Gerrit. Pay attention to your emails for the test results from Jenkins. If you need to troubleshoot a failure, do so by looking at the Jenkins console logs. Ask for help in the #openstack-infra IRC channel if you're not sure what something means, and try to provide as much information as possible including a link to the change in question and specific error messages you think are the problem. For longer pastes, use paste.openstack.org instead of cluttering up the IRC channel. Check if the failed tests are "non-voting" before digging into things. If the tests are "non-voting" their failure doesn't matter with regards to your patch being accepted because they may have known issues.

If it looks like a fluke, you can re-run the tests by adding a comment with the text "recheck". All automated tests must pass for code to be merged, and typically reviewers won’t even begin reviewing a new patch until the checks have passed.

Step 9: Get Your Patch Reviewed

Once the test have passed, you'll need to get your change accepted by 2 core members in order to get it merged. If your patch has been reviewed and says "Needs Workflow" it means another core needs to approve it before it can be merged.

To find out who the core members are that can +2 or approve your patch, visit the project's page in review.openstack.org. Click on "Access" at the top of the screen and then click on any of the project-core links to see a list of people. You can add names as Reviewers to your change from your change's url or you can ask in #openstack-infra if those individuals could review your change. Not everyone monitors their emails if you just add them as a Reviewer.

If you get comments and people want you to make changes, generally discussion should happen in Gerrit so there’s some history tied to the change. However, if you’re still uncertain or need more back and forth than Gerrit can offer, feel free to discuss the change publicly in the #openstack-infra IRC channel, or privately with the reviewer on IRC if you’re not yet comfortable speaking in the public channel. Many people in the community are very passionate about wanting the best for OpenStack, and they have strong opinions as a result. If you don't agree with specific feedback because you feel you had good reasons for why you did a thing the way you did, feel free to politely share those reasons. Chances are you'll have a nice discussion and you'll both learn something ;) If for some reason you're not satisfied with the result of the discussion, feel free to seek a second opinion from another core member. It's ok if you get overruled, don't be frustrated. Just try your best to understand why they want it done that way and move on to the next thing. When you're working with groups, sometimes you have to pick your battles. Once you get more established in the community and understand more of the context, you might better understand their reasons or you can be the core person making those decisions. It's all just part of being in a community with other humans :)

Once your patch has been approved, it will be automatically merged and you'll get an email notification (if you've opted for those)!

Step 10: Do the First Patch Dance!

You can watch your progress as an OpenStack contributor at Stackalytics!

Useful Links

• OpenStack Developers' Guide - http://docs.openstack.org/infra/manual/developers.html
• OpenStack Docs Team "First Timers" Guide - https://wiki.openstack.org/wiki/Documentation/HowTo/FirstTimers
• Virtualenv - https://virtualenv.pypa.io/en/latest/
• Smartcd - https://github.com/cxreg/smartcd/
• OpenStack Git Repositories - http://git.openstack.org/cgit/
• DevStack - http://docs.openstack.org/developer/devstack/
• OpenStack Python Developers' Guide - http://docs.openstack.org/infra/manual/python.html
• Git Commit Messages - https://wiki.openstack.org/wiki/GitCommitMessages
• Paste - http://paste.openstack.org
• Stackalytics - http://www.stackalytics.com

Bike to Work Jeans

A lot of options are entering the market for bicycle commuter clothing, including Levis, Betabrand, ... The main problem I have is the jeans are very low rise and don't come in sizes that fit me. Recently I was browsing L.L. Bean and stumbled across the most amazing thing... "active" jeans! L.L. Bean does a great job of accommodating a variety of shapes and sizes and their quality is pretty good.

I particularly love L.L. Bean jeans because they are made for Pear shaped women. The legs are usually a bit more generous than typical jeans and the waists are more nipped in with a higher rise.

The Performance Stretch Jeans are skinny jeans but they are more of a comfortable slim fit rather than a super squeezed in fit. If you have chunky legs due to either muscle or fat, these will fit you. If your waist is smaller than your hips by a lot, these will fit you.

Finally, they are made of a special performance material and are incredibly durable. I've worn them 3-4 times a week non-stop since I got them earlier this year and they have shown zero signs of wear besides a bit of fading. They also dry faster than regular jeans (although I would not call them "quick drying"). If you tumble dry low for 30 minutes after a spin cycle, they'll easily be done within that time. Hang drying, they are dry within 1-2 hours depending on humidity and air flow. If you're washing in the sink, they'll be dry overnight if you can wring them out with a towel. If you sweat in them or get drenched in a downpour, they'll dry within an hour.

L.L. Bean Performance Stretch Jeans are great for biking!

Another feature I love are the little reflector tabs on the back belt loop and the cuffs!

I have taken these on short trips travelling and I've been impressed at not only the overall durability but their tendency to not retain oders.

Sizing tip, if you want a really tight, skinny fit, then order down. Otherwise, you can safely go by the size chart. It is true to size and if your hips are a little wider than what's listed, don't worry about it because they are stretch jeans and they will stretch out a little bit. My measurements are 34" high waist (38" low waist), 48" hips. No size 20 was available but it would have been too large in the waist. The size 18 fits me perfectly.

L.L. Bean Performance Stretch Jeans, Back View, size 18

L.L. Bean Performance Stretch Jeans, Front View, size 18

If you're looking for an all purpose workhorse jean, something you can do active things in, travel, and still look nice in, I can't recommend these enough!

Grain Free Cupcake v1

When I went off Paleo last year, I learned a lot about Gluten Free baking and even made a pretty decent sandwich bread thanks to America's Test Kitchen. Gluten Free cooking is rife with processed foods and additives, but there's some sound research on the science of baking and how to mix and match gluten free substitutes to make it work. America's Test Kitchen Gluten Free Cookbook is among the best out there, not so much for the actual recipes, but for understanding how the ingredients work together.

If you're going to engage in Grain Free baking, I find that gluten free recipes are much easier to start from than recipes depending on wheat flour. Besides ATK's book, I have this book by Bruce Fife called "Cooking with Coconut Flour" which I use as the basis for most of my experimentation with baking. I don't think I've yet followed a recipe exactly, but that's kind of my style of cooking.

The basic components you're looking for are:

• insoluble fiber for bulk
• starch for lightness
• protein for rise and browning
• soluble fiber for binding

The problem with just these components is that they don't quite mimic gluten. You can bake pretty good things with just these, but they won't really rise well or have a nice crumb. Despite the soluble fiber, they will still be a bit dry and crumbly. Many Gluten Free recipes use gums as an extra binder to more closely mimic the structure gluten builds which improves rise. These gums tend to cause me a lot of gastric distress even in small amounts (and I'm not alone in this). But wait! There's hope!

Recently I've been reading on many gluten free cooking blogs that potato flour can substitute quite well for the gums typically used in gluten free baking to replicate gluten's elasticity. I've actually used boiled potato with wheat flour in the past to improve the texture of the final bread. It makes the texture a bit more soft and tender, which is something non-grain flours definitely need. The advantage of using potato flakes (ie, instant potatoes) or potato flour (different from potato starch) is that you don't get water weight variations like you do with fresh potatoes. That being said, fresh potatoes are cheap and don't require a trip to a special grocery store. And, chances are there aren't any additives other than dirt.

I'd been wanting to try making cupcakes for some time but hadn't had the right inspiration. I appreciate coconut flour, however I'm not the hugest fan. It tastes weird, it's dry and crumbly and you have to use crap tons of eggs to even make anything decent with it. Not to mention all that insoluble fiber just goes right through me. After reading about potato flour, I got inspired. So tonight I tried my first attempt at grain-free cupcakes, and I'm quite happy with the results!

If you are looking for a more frequent treat, you can skip the icing and use minimum amount of honey (2T). For a special celebratory treat, I don't see anything wrong with a little refined sugar frosting soaked in butter.

Paleo Cupcake v1 - The secret ingredient is POTATO!

Grain-Free Cupcake v1

The Wet:

2 eggs

2 - 4T honey

3 T melted butter

1/4 t sea salt

1/2 t vanilla

2.5 oz or 72g of cooked white potato (floury or starchy, doesn't matter)

The Dry:

2 T coconut flour

1 T tapioca flour

1/4 t baking powder (homemade with 1t baking powder + 2t cream of tartar)

Beat the eggs and honey together, then add the butter, salt, and vanilla. Mix until well blended. Press the cooked potato through a strainer or ricer and mix in until smooth.

In a separate bowl, mix together the dry ingredients and sift them into the wet ones. Mix until you have a wet, smooth batter.

I used a 12 count USA Pans muffin pan. Line the middle 2 rows (inner 6 spots) with greased paper cups and pour the batter so it fills the up about halfway. You should have enough for 6 cupcakes.

Bake in the upper part of the oven at 400 degrees Fahrenheit for 15 minutes. Let cool completely before icing or eating so the starch has time to set. These should rise to the top of the muffin compartment.

Buttercream Frosting

Any good cupcake of course needs frosting and if you're looking for a simple buttercream, my recipe is adapted from King Arthur Flour's. Granted the use of refined sugar may be offensive to some die-hards out there, but I've always considered cupcakes more of a rare celebration item and not so much of an every day of the week phenomena.

.75 ounces butter

.25 ounces lard

1/4 t vanilla

tiny pinch of salt

4 ounces confectioners sugar (I use 365 Organic with tapioca flour as the starch)

2T milk or cream

Let the fats get to room temperature, then beat them until creamy. Add 2 oz of the confectioners sugar and beat until blended, then mix in the salt and the vanilla. Add 1 T of the milk, then alternate adding in a bit of the confectioners sugar and milk until it's smooth and creamy. I put mine in a plastic sandwich bag and snipped off the corner for an instant mini pastry bag. In my photos, I've added a bit of lime zest.

If you want to add food coloring, I use gel food coloring. For a more natural alternative, you could maybe mix some beet juice and gelatin, but I haven't tried it myself ;) 

Outer crumb of Grain-Free Cupcake v1

Inner crumb of Grain-Free Cupcake v1

This cupcake is a little dense but really not too bad. The most important thing it stays together really well and actually peels off the paper almost like a regular cupcake. The coconut flour makes it a little grainy, possibly soaking the flour for a bit before putting it into the oven could help with that. I also realized I screwed up my homemade baking powder ratio when I was writing this blog and ended up doing 3 baking soda to 1 cream of tartar. I'm going to bet that these would have risen better if that had actually been right ;)

Final Thoughts

Potatoes have been a controversial subject in the Paleo community. My stance on it is my diet is not a religion and I base my diet decisions on science. Do YOUR OWN elimination/reintroduction experiment to determine what to include or exclude from your diet. There's a lot of pseudoscience when it comes to nutrition and the Paleo community is no exception (serious Snake Oil Peddling, amirite?). Recent studies have shown a diet high in starchy tubers does not necessarily lead to diabetes and obesity. The energy in starchy tubers is intended to store food and water for a plant for a long period of time, whereas a grain's starch is intended to be rapidly converted to sugar to feed a germinating seed. Ever done any home brewing? Ever tried it with potatoes? Because potato starch is intended to be released slowly over a long period of time, fermenting grain works a little better. So, logically, it seems reasonable to conclude our bodies would process the starches differently as well.

But really the ultimate test is your own reaction to the foods that you eat. In my case, I ate 1 small white potato. One hour later, I checked my blood sugar - 105 (resting is 90). Within my normal post meal range. No digestive issues or reactions. So much for that whole "glycemic index" nonsense.

Cream of Mushroom and Leek Soup

My husband really likes Cream of Mushroom Soup. I think it's ok but it's not my favorite. I usually like to add some oomph to it to make it more interesting. Leeks will give the soup a nice sweetness, so make sure you saute them so they are sufficiently caramelized.

Mushroom soup isn't particularly photogenic

Ingredients

4 T Butter
16 oz Mushrooms, separated into caps and stems, caps sliced
2 medium or 1 giant leek, chopped
1 quart (give or take) Chicken Stock (or other light stock)
8 oz Heavy Cream

Seasoning

Salt and Pepper to taste
Nutmeg (to taste)
1/4 Lemon (for juice)

Roux

4T Butter
1/4c + 2T Flour

Preparation

1. Saute the mushrooms and leeks in the butter for about 15-20 minutes on low until they start to emit an aroma and they are soft and shiny.
2. While the mushrooms and leeks are sauteing, add the mushroom stems to the chicken broth and simmer on medium. If using unseasoned or packaged chicken broth, add some aromatics at this time to boost the flavor.
3. Add the chicken stock to the mushrooms and leeks and simmer for 1 hour.
4. In a separate pan, make the roux using the flour and butter. Bring the soup to a boil then add the roux to the soup.
5. Boil the soup for 5 minutes, stirring constantly, until thick
6. Turn off the heat and add the cream. If the soup is too watery, you can turn the heat up to high and boil it until it is thick enough, just make sure you stir it frequently to prevent the bottom from scalding.
7. Add seasonings and lemon juice.

Cooking Tip

Lemon or Lime Juice is my secret ingredient. The citrus adds a wonderful brightness to dense dishes and often it's not obvious that there was lemon specifically added to the dish. I usually use a little lemon juice in stews and vegetables. Orange juice and zest works great with squash. Lime is especially nice with coconut milk or bean-based soups.

If you're worried about seeds or pulp getting into your food, you can squeeze the lemon or lime through a strainer. I do not recommend pre-squeezed juices, they aren't as good as the real deal and getting it straight from the fruit really isn't hard.

For a great refreshing soda-like beverage, squeeze 1/4 to 1/2 of a lemon into a 16 oz glass and top up with cold seltzer water. You can also add the same amount of juice when you refill your water bottle for the gym or what have you with filtered water. I often put my daily dose of liquid magnesium supplement into this mixture because the lemon covers up the bitterness!

Bauernfrühstück

I think almost every culture has some form of a basic peasant dish that consists of potatoes, onions, eggs, and meat. In Spain you've got the Tortilla, in Italy it's the Frittata, in Ireland it's Colcannon, and in the USA we technically call it a Hash but restaurants have it on the menu as a "Skillet" or "Scramble".

In Germany, this dish is called Bauernfrühstück (bow-urn-frew-shtewck) or "Farmers' Breakfast". I learned about this dish from my German husband, did a little research online and in my German cookbooks, and added my own adjustments based on my own preferences and cooking experience. I've even had it a few times in Germany, but of course we both like mine better than anything we've had in a restaurant ;)

Bauernfrühstück!

For this recipe, I'm sharing my proportions for 2-3 people. 2 big eaters, 3 light eaters, and if you just added an extra egg, this would work for 4 light eaters without any other adjustments. This isn't the kind of recipe you need to follow exactly, you can easily adjust proportions based on your own preferences and what you have on hand, and the result will be perfectly fine.

Ingredients

2 T butter
1 T lard or bacon dripping
2-3 rashers of uncooked thick cut bacon, chopped (see substitution note)
1 onion, halved and sliced
1 potato, quartered and sliced (thinner cooks faster), raw or cooked
1/4-1/2 cup chopped ham (see ingredient note below)
3 eggs
Paprika, 1-2T or less if you don't like paprika that much
Salt
Pepper
Parsley leaves, chopped, 4-5 leaves
Chives or the green part of scallions, chopped, about 1/2 T
1/4 mild flavored cheese (like Edam), shredded (optional), because I'm American I like cheese on everything, but this is definitely NOT a traditional German ingredient!
Sour cream or Creme Fraiche for serving (optional)

Substitution Note: If you don't have bacon on hand, increase the lard or bacon dripping to 2T (really it's enough fat in the pan to keep things from sticking, so use your best judgement).

Note about Ham: You can use thin sliced deli ham for this, but if you do, you'll want to have some nice thick cut bacon as well or else it won't have a good depth of flavor. If you use thin cut ham, about 1/4 cup along with the chopped bacon should suffice. If you don't have bacon, this is a great way to use leftover ham from a big ham roast. The ham roast will have a nice depth of flavor that holds up well such that you don't need to also add bacon. Again, use your judgement and experiment according to what you have on hand. This is peasant food after all :)

Instructions

1. Saute potato, onion, and bacon if using over medium-low heat in the fats and add 1T of paprika plus some salt and pepper. If you are using ham from a roasted ham, add that now. Saute until the potatoes are soft. I recommend using a cast iron skillet and covering the skillet if the potato slices are on the thicker side. This will take around 10-15 minutes. Check the potato slices for doneness, they should be soft. If you are using cooked potato, saute until heated through, this should only take about 5 minutes.

2. Beat the eggs with 1T paprika, salt, pepper, chives, and parsley. If you are using thin sliced deli ham, add it to the egg mixture. Pour over the potato and onion mixture in the skillet. Cook until the bottom of the eggs look mostly set. You can cover it during this cooking period, I find it helps the eggs to cook more evenly.

3.  Use a spatula to scramble the eggs around a bit and get any stuck bits off the bottom of the skillet. At this point, you can sprinkle the shredded cheese on top of the egg and potato mixture. Again, you can cover it or not, your choice really. I find it helps things to cook through a bit faster, but that's me.

Cheese is definitely not part of a traditional German Bauernfrühstück!

4. Once the egg mixture is fully cooked (and cheese melted if you added cheese), turn off the heat and serve it up!

I like mine with a dollop of sour cream or creme fraiche, but it's perfectly acceptable to just eat it plain, or even just sprinkle a bit of parsley over top. If it tastes a little flat, try a few drops of fresh lemon juice.

I make this every weekend, sometimes both Saturday and Sunday, or if I'm working from home and have the time or energy I'll even make it then. It takes about 30 minutes if you're using raw potatoes, but with cooked it takes about 10-15. Years ago in college, I made Spanish tortillas all the time because they were easy, cheap, nutritious and satisfying, and they kept well in the fridge. Bauernfrühstück does not keep quite as well but if you keep it at room temperature and your kitchen isn't too warm, then you can eat it the next morning. Putting it in the fridge will alter the starch in the potatoes, making it less desirable to eat.

Thai Curry Un-Recipe

Whenever I order Thai Curry from most restaurants, I'm usually disappointed. It's typically too sweet, not spicy enough, watery, oily, and lacking in flavor. I've never been to Thailand and experienced the real deal, but I do believe it's possible to make something more satisfying in ones own kitchen.

You really don't need any special exotic ingredients to make the basic recipe and you can even get decent curry paste online. This is a very quick dish assuming you have the stock already made. I have used boxed chicken broth in a time crunch and it has worked just fine.

Quick disclaimer: My style of cooking is to casually follow a sort of outline of ingredients and instructions, improvising depending on my own preferences and what I have on hand. Hence the Un-Recipe in the title :)

Homemade Thai Green Fish Curry! It doesn't look like much, but trust me, it's tasty!

About the Ingredients

I'm going to talk a little about some of the key ingredients and then provide a recipe and instructions for how to make the curry.

Curry Paste

I highly recommend making your own curry paste if you have access to the raw ingredients. If you can't find things at Asian markets or grocery stores like Whole Foods, you can usually find them online. The flavor of your own curry paste will always be better and fresher than any curry paste you purchase pre-made. The other advantage of making your own is you can manage the spice level yourself. For instance, my husband hates spicy, so if I make curry using the ready made curry paste, he can't eat it. I've tried a few different recipes throughout the web and this one is my favorite. Note that Galangal is a type of ginger and if you can't find it you can try to substitute young ginger if you can find it. Galangal has a very floral sort of taste to it and is milder than ginger. You can try using reconstituted dried galangal if you can find it online.

If you don't have access to the raw ingredients, or are just lazy (like me sometimes) the best ready made curry paste is Mae Ploy. I've seen this brand used in Thai restaurants in Seattle (that I actually liked their curry), so if you want to reproduce that kind of flavor without making your own paste, then this is the brand to use. I personally prefer the Green Curry Paste, but the Panang comes up close second. I have tried the Red and I'm not a huge fan, but it's not awful. Just not my favorite. The other reason I recommend Mae Ploy is that there are no additives. The items on the list are exactly what you would put into it if you were making it yourself, there's no special preservatives or anything like that.

My advice is to use Green curry for poultry and fish and to use Panang curry with red meat.

Coconut Milk

Most coconut milk is canned and has added preservatives and stabilizers. Making your own coconut milk is the most economical and of course produces the best quality result, however it's pretty labor intensive. And given that the stuff doesn't really preserve well, you'd be stuck doing it each time you want to make curry. I mean if that's your thing, and some days, it's totally mine, then go for it. You can even make coconut flour with the leftover desiccated coconut! It's definitely the way to go if you have the time and want to save money.

That being said, it's tough to find additive free coconut milk. Fortunately I have found it online! I buy the Arroy-D in the 8 oz size and find one or two is enough for most of the recipes I make. I normally get the 6-pack but I just noticed they have 12-packs now! Sweet!

Once opened, it does not freeze well and will go sour within a day or two (just like real coconut milk.. cuz this is.. real coconut milk). If you don't want to use a whole carton, you can always give any leftovers to your dog, they love that stuff!

Kaffir Lime Leaves

These are strictly optional, you can use lime zest if you don't have them, but the flavor will be slightly different. If you do find them in a grocery store (again, sometimes places like Whole Foods have them if your Asian grocery store doesn't), you *can* freeze them. I usually buy a bunch and store them in the freezer since they aren't delicate and don't take up much space.

Fish Sauce

I use the Red Boat brand because it's very high quality and doesn't contain anything beyond salt and rotten anchovies. I use Fish Sauce to make my own HFCS-free Worcestershire sauce and often throw it into stews for a little Umami. Trust me, get this stuff, you will end up using it for all kinds of things.

Sugar

When sweetening your curry, Sucanat is the best thing to use, but you can use brown or turbinado sugar if that's what you have. Thai cooking traditionally uses palm sugar or Jaggery (a blend of palm sugar and cane sugar) or coconut sugar (not very environmentally sustainable). I used to use coconut sugar a lot and it worked really well with the Thai curry but after reading about how it's being harvested and how it destroys coconut crops, I decided not to use it. Sucanat is the most minimally processed sugar you can buy, it's literally just evaporated cane juice with all of the molasses and minerals intact. The point is, you want to use sugar that has a molasses flavor to really give the curry a well rounded sweetness.

The Recipe

This is for 1 quart of stock, which ultimately makes 6 1-cup servings of the curry.

2 T butter, ghee, coconut oil, or other saturated fat

About 3 cups of chopped vegetables, whatever you have on hand, be creative!

The photos here used: 1 medium sized potato, 1 very large carrot, 1/2 onion, 4 oz green beans

You can also use zucchini, eggplant, bell pepper, bamboo shoots, sweet potatoes, just think about what's typically in Thai Restaurant menus for ideas.

About 2 cups of chopped meat, raw or cooked

1-2 T curry paste (depends on how strong you want it)

2-4 cups of homemade stock, this can be chicken, fish, duck, whatever. Chicken is the most universal and goes with the largest variety of meats. In this case I used fish stock because I had baked a whole fish, made stock from the leftover bones, and had a bunch of fish meat leftover. Again, use what you have on hand and be creative. If you plan to thicken the stock, then use more stock, otherwise use less.

12-16 oz of coconut milk, add this to taste. I often use a lot of coconut milk, but you can use less depending on your preference and how thick you want your curry.

Salt, to taste

Fish Sauce, to taste

Sucanat, to taste

2-4 Kaffir Lime Leaves, optional

1/2 of a Lime (for juice)

Optional: 1-2 T flour or 1/2-1 T rice flour for thickening Traditionally you don't thicken Thai curry, but if you want to you can. I personally like it thicker so I make a roux. It's kind of fun because then your curry almost looks like Japanese curry before you add the coconut milk. Use rice flour if you aren't a fan of wheat, I recommend sprouted brown rice flour if you can find it, it's not quite as grainy and has enough protein in it to brown like wheat flour.

Instructions

1. Sautee the long cooking vegetables (things like potatoes, eggplant, carrots) in the fat over medium heat until they are glistening (5-10 minutes). If you are using a slower cooking raw meat, then add it now (chicken, beef, duck, etc).

2. If thickening add the flour and stir until the flour has absorbed enough fat that it won't clump (about 2 minutes)

3. Add the curry paste and make sure the vegetables are well coated.

After sauteing onion, potato, and carrot in butter, add some curry paste

4. Add the stock and simmer until the vegetables are cooked through. You can either simmer on high to reduce the stock to make it thicker, or boil the stock for 5 minutes until thick from the added thickener and then turn down the heat.

The fish stock came straight from the fridge so it was very gelatinous!

5. While the stock is simmering, add the kaffir lime leaves, Fish Sauce, and Sucanat. You can add more later so be conservative, this is just to get some flavor cooked into the vegetables.

6. Once the curry has reached the desired thickness, add coconut milk and adjust the seasoning by adding salt, fish sauce, and/or sugar. Simmer for about 5 minutes.

I use Arroy-D 100% coconut milk with no additives

7. Add the quick cooking vegetables and the chopped meat (if cooked or quick cooking like fish).

Adding the chopped up cooked fish and the frozen green beans

8. Squeeze the juice out of the lime half, and adjust seasoning if needed. Serve over rice, rice noodles, or just eat it like soup!

Cooking Notes

I learned an interesting technique for chopping carrots from a Japanese cookbook. This technique has you create sort of rounded wedges as you chop the carrot. I use this method when chopping carrots for stews because I find the soup clings to the carrot and it is much more enjoyable to eat in this shape.

To cut a carrot in this manner, you just cut a wedge off the end as you roll the carrot.

Chopping a carrot, Japanese style