Saturday, November 16, 2013

First Nighttime bbq!

Adding a little extra warmth to the already unusually warm San Franciscan Veterans' Day weekend evening with Sonrise folk and other friends.

Sunday, October 27, 2013

FlightGear Rembrandt: HAF -> SFO

The windy walk back from work during dusk reminded me that I do in fact live in San Francisco. And although my work schedule doesn't allow me to get out of town all that much, fortunately my flight simulator tries to fill in the gap. A couple days ago I decided to flip on the new "Rembrandt" setting in the flight sim, FlightGear, and I found myself sinking into an entirely new cockpit, as if it were the real thing. As I took a short hop tonight from Half Moon Bay Airport to San Francisco International, the airport lights flickered just beyond the cabin glow of my cockpit. And just to make things feel truly real, immaculately surreal, the sim's real-time weather brought me from the fog of HAF airport to a crosswind landing at SFO.



A couple weeks ago, I did actually get to venture inside a cockpit at SFO, this one a 737-900. My brother's friend from church gave us tickets to a United Airlines show, where I got to take a snapshot of my niece in her future profession (or so we hope!).


To my amazement, as soon as I took these snapshots from my Android phone, Google identified them as coming from a 737. It even guessed a 737-800--pretty close! Now if FlightGear could get its planes identified by Google as "the real thing," that would be rather impressive. And judging by the rendering in Rembrandt, I think that day may not be so far off.


In any case, as much as I enjoy the relaxing realism of a virtual flight after a long day, I'm thankful for the real pilots keep our real bodies safe in the air, day after day after day.


Monday, October 14, 2013

Installing Matplotlib and Mayavi on Mac

I had lots of fun with Matlab in grad school, but I always had in the back of my mind that at some point my educational subscription would run out. Thus started my slow search for alternatives, preferably open source alternatives with combined graphing capabilities.

I also had lots of fun with Python, so my hunt brought me to the Matplotlib, a Pythonic scientific graphing tool. While Matlab has a single application (plus many toolkits) to install, the Pythonic solution has no less than Python + Numby + Scipy + Matplotlib to get things rolling. Installing Matlab is as simple as starting and installer and letting it do the work, whereas installing Matplotlib was another story. Allow me to document that story here in the likely chance that I'll otherwise forget it.

Matplotlib can be installed through a single solution provided by a company named Enthought, but my hope was to install everything using standard open-source installers. Here's my setup and the specific requirements I was aiming for:
  • Mac OS 10.8
  • Homebrew for package management
  • Python installed from Homebrew rather than from Apple XCode
  • Virtualenv to isolate a "scientific Python" environment
  • Matplotlib as a primary graphing tool
  • Mayavi as an alternative 3D graphing tool
Installing Homebrew

Back in college I became an avid Linux user, and part of both the joy (and occasional angst) of Linux was the package management system. rpm, yum, and apt-get all provided ways to keep software up to date without having to check individual sites for updates.

When I had to use Windows more often, I found the Cygwin system a boon both for accessing all my Linux/bash scripts on Windows as well as keeping everything up to date. At some point, I did install Matplotlib via Cygwin but, alas, did not document instructions and have since forgotten the details of the process.

Now that I'm using a Mac, I found Homebrew to meet all my package management needs. Installing it was a matter of running a simple command (scroll to the bottom of that page). Keeping it up to date involved brew update; brew upgrade, and that was that.

Installing Homebrew-based Python

One of the tricky aspects of Homebrew is that Python can come from multiple locations. Apple XCode already includes Python, so by default Python will run, albeit an older version. Installing the Homebrew-based version can be a help as it brings a newer version and also allows customization so that it will work with a wider variety of packages. The Homebrew-based package also apparently brings pip with it for additional installations.

To install Python 2.x, I used the command:

brew install python --with-brewed-openssl
Python 3.x could also apparently be installed side-by-side with Python 2.x by replacing "python" with "python3", although I haven't yet tried out 3.x.

I did come across some errors, most likely based on a prior installation of pip outside of Homebrew. This SO answer and blog post helped me revive the installation.

Installing pip and virtualenv to install more packages

Now that we've used one package manager, we get to use another package manager, this time pip, a Python package manager. And the first package to install is a tool to manage multiple sets of packages. Talk about recursive package management, huh?

Following this guide, my first step was to install virtualenv, which allowed me to isolate the subsequent packages in case I later wanted to install a different, potentially conflicting set of packages. A side benefit is that if I made a mistake during installation, it was a matter of deleting a virtual environment rather than trying to clean up my entire system.

I used this command to install virtualenv, using pip provided by the Homebrew-based Python.

pip install virtualenv
To keep my virtual environments together, I made a separate directory and then created a new environment, which is started through the activate script.

mkdir ~/virtualenvs
cd ~/virtualenvs
virtualenv pystat
source pystat/bin/activate

Installing Numpy + Scipy + Matplotlib

And now, time to install what we set out for! First, be sure to deactivate the virtualenv session to install from within a normal shell session prior to running further brew commands.
UPDATE (9/5/2016): Looking back through this post, I think that the pip commands still need to be within the virtualenv to make isolated environment relevant. Also, I'm not sure how isolated this entire configuration is since the Homebrew installations occur outside of the environment, and updates appear to break the installation. For example, a recent incompatibility with vtk appears to have rendered my installation unusable at least for now.
deactivate
pip install numpy
brew install gfortran
pip install scipy
brew install freetype
pip install matplotlib

Here's a screenshot of visualizing football models with Matplotlib.


Installing Mayavi

While testing out Matplotlib for 3D histograms, I came across a few rendering issues and this faq from the Matplotlib site itself that suggested using Mayavi as a more powerful 3D engine. Installing Mayavi was no laughing matter, but it did work in the end, particularly with the help of this guide.

Installation involving tapping another Homebrew repository, homebrew/science,

brew install cmake
brew install qt
brew install pyqt
brew install --python --qt vtk
brew install wxmac
pip install configobj
pip install envisage
pip install mayavi

During the final step, I was initially unable to install mayavi because of an error in finding the vtk package, despite having successfully installed it. Apparently vtk gets installed in the Python system site package and is not accessible from within the virtualenv. My workaround was simply to turn off the switch that was preventing access to these packages while in a virtualenv:

rm ~/virtualenvs/pystat/lib/python2.7/global-site-packages.txt
pip install mayavi

Mayavi is installed!

UPDATE (2/24/2014): While trying to update Homebrew today, I ran into a few errors. The first error stated:
vtk: Unsatisfied dependency: matplotlib
Homebrew does not provide Python dependencies; install with:
  pip install matplotlib

Matplotlib was installed within the virtualenv and does not appear to be visible to Homebrew. I ended up having to activate the virtualenv where matplotlib was installed and running the Homebrew update from there. vtk did show up in the Homebrew global Python site-packages directory (/usr/local/lib/python-2.7/site-packages), even when installed from the virtualenv.

I also encountered a error on updating wxmac:
/usr/local/Cellar/wxmac/3.0.0.0/include/wx-3.0/wx/strvararg.h:25:14: fatal error: 'type_traits' file not found
    #include <type_traits>
             ^
1 error generated.
error: command 'clang' failed with exit status 1

Apparently this error has been reported and commented upon extensively several months back with a workaround. The solution was to run:
brew edit wxmac
and add the line:
ENV.append_to_cflags "-stdlib=libc++"
to the 'cd "wxPython" do' block (currently line 39, where the line can be added after the existing append_to_cflags call).

But alas, it would not run. I received the error message:

This program needs access to the screen.
Please run with a Framework build of python, and only when you are
logged in on the main display of your Mac.

which fortunately had this handy SO solution with a script to run Python from within the requisite environment (or see this alternative script). Here's a screenshot of visualizing that same football model but with Mayavi.


Now I'm enjoying both Matplotlib and Mayavi home-brewed in a cozy virtual environment. With a real coffee in hand, of course.

Friday, September 27, 2013

Android-x86 4.3 on VirtualBox with Google Play Services

I've run Android on VirtualBox over the past couple years (2.3, 4.0 part 1, 4.0 part 2) as it provided a considerable performance improvement over the standard emulator. After I picked up an Ivy-Bridge-based Mac Mini, the sheer performance improvement over my prior machines made up for much of the slack in the emulator and allowed them to be a pretty decent and simpler alternative to the VirtualBox solution. Recently however I did find another excuse to re-explore Android on VirtualBox and found a couple pleasant surprises.

Out-of-the-box 4.3 from Android-x86

I learned that Android-x86 posted a build of 4.3 shortly after its arrival, and I decided to give it a shot. Unlike prior builds of earlier Android versions, this build worked right out of the box. There's really not a whole lot of configuration details to add from my prior experiences. Here's a screenshot of a basically un-customized install ticking its way onward to 2 weeks of uptime.



I used the same settings that I had used for Android 4.0, except that I happily managed to get away with using only 256 MB of RAM. There was no need for special partitioning for an SD card, and customizing screen resolution was similar to my prior experience except that I didn't have to adjust the DPI setting.

Google Play Services

An additional motivation to retry Android on VirtualBox was to see whether Google Play Services might work better on it. I recently integrated Google Play Game Services into tXtFL Mobile, and one issue that cropped up during testing is that the standard emulator's version of the Play Services is baked into the emulator image so that it can't be updated separately from the entire emulator image.

Where this became a problem was when a version of the Google Play Services SDK was released that was incompatible with the emulator's version. Android on the emulator gave an error message to the effect, "Google Play services out of date.  Requires 3159100 but found 3136130," and unfortunately that meant waiting for another emulator release, which took almost a month to fix after it was first reported and reared its head again briefly awhile later although has now been fixed.

The Android-x86 version thankfully includes the Google Play Store, and what that means is that unlike the standard emulator, Android-x86 on VirtualBox can install updated versions of Google Play Services. After initially installing Android-x86 4.3 on VirtualBox, my app promptly crashed while attempting to sign into Google Play. After waiting awhile and rebooting, however, I found that Play Services had updated itself, and subsequent sign-ins worked without a hitch.

Play Services has continually kept itself updated since then, and here's the latest version, with a number higher than that on my phone or the emulator (834000-30). The version number itself is the same though (3.2.65).


Now I just have to wait for Android-x86 to release a 4.4-based build.

Sunday, September 15, 2013

A's vs Giants in 2013

Last season, the A's and Giants were in the race right up to the end. This season, they've taken very different routes, and this snapshot from today's out-of-town scoreboard symmetry pretty much sums up the season:


SF was actually up 1-0 until a bases-loaded, bases-clearing double put them back in the hole. The A's are trying to preserve their lead in the game and widen their lead in the divisional race with the Rangers.

Wednesday, July 03, 2013

"Keep your eyes on the ball"

This past Friday I attended an A's-Cardinals game at the Coliseum, thanks to a 2-for-1 ticket my buddy picked up. I was testing out the panorama function on my new HTC One and just so happened to catch the first single of the game. The panorama setting appears to use the exposure settings from the first frame, so I swept from right to left--in this case, from batter to outfield--which turned out to double as a way to keep my camera's eye on the ball from the batter to the outfielder. Here you can see Coco Crisp's swinging, the 3rd baseman's lunging, and the left fielder's fielding the ball all in one frame, as if all took place simultaneously. If you look closely, you might even see multiple versions of the ball coursing their way to the outfield.


Sunday, May 19, 2013

"Save both yourself and your hearers"

Today's sermon touches on 1 Tim 4:11-6, where Paul exhorts Timothy to preach so that he might save both himself and his hearers. The question proposed in the sermon is how he can save anyone when we know that only God, only Christ Himself, can save us? The answer proffered is that we save by preaching the very Gospel that saves, that in a sense we do not save, but we provide the message that saves.

The challenge with this interpretation however is that the verse specifically says that we do in fact save, not merely that we are a conduit for salvation. God is certainly the one who works in us to save, but somehow Timothy is the direct agent to "save both yourself and your hearers." So how then do we as mere humans actually perform salvation? I think that the key is to ask ourselves, from what are we saved? In many cases the Scripture points to salvation from hell and sin, a very spiritual salvation that only God can enact in us. If I shared the Gospel to someone who then chose to follow Christ, I would never say that I saved him, but that Christ saved him.

We know however that there are many other things from which we can be saved. In this passage, Paul appears to refer less to eternal salvation and more to the practicalities of daily life and ministry. He speaks of the challenges of service and of preaching to guide people through the daily struggles of life. He may therefore be referring to the very power and responsibility we have to save people not only from walking into gross sin, but also from all the senseless trials, conflict, and even simple stupidity that we have all committed. Through the experiences that Timothy has suffered and the maturity that he has gained despite his youthfulness, he can share with them the strategies that he has gained and the Scriptures that he has learned to help save them from falling into the mistakes that he has committed himself or as a leader has seen others make.

By helping to "save both yourself and your hearers," we can also exemplify on a daily basis God's eternal salvation of us. By saving people from all sorts of practical, visible troubles that we might otherwise find ourselves in, we can more readily see and more deeply appreciate the internal and eternal salvation that only God has provided us in Christ.

Saturday, May 18, 2013

Migrating from Subversion to a simple self-hosted Git server

I recently migrated a Sourceforge-hosted Subversion repository to a Git repository for one of my open-source projects. I also have a number of private projects holed up on an old Linux server at home and decided that it's nigh time to git them over to Git as well. Now that I have a Mac Mini, I figured that I could take advantage of its Unix backbone to set up a simple SSH-based, self-hosted Git server.

My code will get an upgrade in hardware and version control protocol in one fell swoop. And given the number of minor tricks I had to find along the way, especially with the help of this tutorial, I figured that I'd document them here for posterity's sake.

Exporting the Subversion repository
I again used svn2git to export the Subversion repository to a local Git repository. In my case, I had a multi-project Subversion repo and wanted to export only one sub-project, so I used the "--no-minimize-url" flag. It also happened to be in slightly non-standard layout ("TxtflMobile" instead of "trunk"), which required me to explicitly set the trunk name. The authors flag was as outlined previously.

[on local machine:]
$ mkdir prjdir
$ cd prjdir
$ svn2git http://path/to/repo/subproject --trunk TxtflMobile --tags tags --branches branches --no-minimize-url --username name --authors ~/authors-txtfl.txt --verbose

As the repo was private, I had to specify the username to access the repo. Strangely, the execution hung, and I realized that I was supposed to provide my password even though it didn't prompt me--that is, until after  I provided my password and hit Enter.

To check the export, I ran the "git branch" and "git tag" commands to see the listing of all branches and tags.

Setting up the simple Git server
Looking through Git server options, I initially thought that I would need to download a variety of third-party server packages to set it up. A great many server packages do exist, including 8 outlined here and seemingly several popular packages that have since arisen. Fortunately, one of the easiest of these options is also one recommended in the Git book itself, the SSH-based server that allows access through basic SSH login.

To set up the remote repository on the server, I logged into the server (which happened to be the same computer to which I had exported the Subversion repo). In the System Preferences, I made sure that "Remote login" was enabled, which turns on sshd to allow SSH connections. I also wanted to change the sshd port, which involved editing the /System/Library/LaunchDaemons/ssh.plist file and making sure that my router forwarded this port to my server.

Now that the server was accessible (was, in fact, a "server"), I created a "bare" repository in the location from which I would host the Git repo and into which I would push my local Git repo.

[on server:]
$ mkdir /path/to/hosted/repo
$ cd /path/to/hosted/repo
$ git init --bare

Pushing the local repo to the server
Back on the local machine, I prepared the repo for pushing onto the server. I added the remote location to the repo to indicate where to push up the local repo.

[on local machine:]
$ git remote add origin ssh://name@127.0.0.1:123/path/to/hosted/repo

In my case, I used my username and customized port (eg "123") for login. Note that the "ssh://" protocol is necessary to specify the alternate port. In the process of learning this trick, I had to edit the URL for "origin" by using the "set-url" parameter in place of "add". In this case, I used "127.0.0.1" as the IP because I would be pushing the local repo from same machine as the server.

Once I had set the remote URL, it was just a matter of pushing up the master branch, each additional branch, and all the tags to the server.

$ git push origin master
$ git push origin branch1
$ git push origin branch2
$ git push --tags

Accessing the repo from abroad
Of course, one of the main benefits of hosting the hosting the repo remotely is that one can access it from abroad. Using the "git clone" command to the server's URL allowed me to download it onto another machine.

[on another local machine:]
$ git clone ssh://name@hostname.com:123/path/to/hosted/repo

Another benefit of hosting the repo remotely is for backups. The cloned copies each serve as backups, and the remote repo itself can be backed up from the server. As one still getting used to Git, I immediately made an erroneous commit that I had already pushed to the remote server. While reverting this change can be difficult once on the remote server, in this case I merely had to enter the Time Machine backup software and restore the folder on the remote machine from a preceding backup. This simple solution is of course much more feasible given the simplicity of the project on such a simple server.

Monday, May 06, 2013

Migrating a Sourceforge repository from Subversion to Git

I still remember when a friend at church came up to me and said, "I'm gonna tell 'bout something you're gonna love. It's called Subversion." At the time, CVS had been the main version control system for keeping track of files, and it had revolutionized my way of thinking about backups and record-keeping. But CVS had its limitations, including lack of basic functionality such as renaming files cleanly. Subversion was like "a breath of fresh air," where seemingly all of the limitations of CVS went "poof."

But alas, the world has moved on, and Git has grit its teeth deeply into the stronghold of Subversion. Having built so many projects with Subversion behind the scenes, I was rather reluctant to git over to Git. I knew it was only a matter of time, however, and finally decided that I must give it a shot, starting with one of the several Sourceforge projects I administer.

Sourceforge supports a number of version control systems, including both Subversion and Git. Surprisingly, the documentation for migrating a Subversion repository to Git on Sourceforge was rather sparse, and as far as I can see, there's no auto-migration mechanism to simplify the task. Fortunately, there are many other tools to simplify the process, which I'd like to document for posterity here.

The beauty of the migration is that it doesn't require downloading the whole Subversion repository. The tool svn2git can apparently work on the remote repository to do all the conversion, and then the resulting local Git repository can be pushed onto the Sourceforge-hosted remote Git repo.

On my Mac, I first made sure that git, git-svn, ruby, and rubygems were installed. All the packages must have come with XCode Developer Tools as they were all already installed and ready to go. I next installed svn2git using the simple command:

$ sudo gem install svn2git

I then created a new directory to hold the new repository and entered it:

$ mkdir jaj
$ cd jaj

The simplest command for import, which assumes the standard trunk/tags/branches layout, would have been (for my "jarajar" project with mountpoint "code"):

$ svn2git https://svn.code.sf.net/p/jarajar/code

To map the author names to Sourceforge email addresses, I made a simple authors.txt file of the format:

sourceforgeusername = name@users.sourceforge.net
sourceforgeusername1 = name2@users.sourceforge.net

GitHub recommends import of each project within a Subversion repository to a separate Git repo. In my case, my Subversion repo was laid out for multiple projects, even though it really only contained one. To import only this project, I pointed svn2git to this project folder alone and passed the "--rootistrunk" argument:

$ svn2git https://svn.code.sf.net/p/jarajar/code/trunk/jarajar --rootistrunk --authors ~/authors.txt

Note that in the process of experimenting, I had failed imports that kept on failing, even though the subsequent commands were correct as far as I could tell. It turns out that the import process created a .git folder that had become corrupted and was preventing subsequent exports, even with the correct command. Deleting the .git file fixed these errors to start afresh, and running the commands with the "--verbose" command greatly simplifying troubleshooting.

Once I had imported the repository into my new local Git repo, I simply needed to push it to the Sourceforge-hosted Git repo. After creating an empty Git repo using the Sourceforge project Admin tool, I followed the Sourceforge import instructions for pushing a local repository (conducted from within the same directory):

$ git remote add origin ssh://USERNAME@git.code.sf.net/p/PROJECTNAME/MOUNTPOINT
$ git config branch.master.remote origin
$ git config branch.master.merge refs/heads/master
$ git push origin master

And voila, my Subversion repository popped up in my Git repo, including all historical dates from revisions of old. Had I imported the entire project, maybe I would even see that venerable commit message, "New repository initialized by csv2svn." And back then, that had to be done by a Sourceforge engineer.