Information Processing

Pessimism of the Intellect, Optimism of the Will     Archive   Favorite posts   Twitter: @steve_hsu

Monday, September 22, 2014

Piketty on Capital

Piketty on EconTalk (podcast) -- a lively discussion between Russ Roberts and guest Thomas Piketty. See earlier post here.
Piketty: ... to summarize very quickly our conclusion, we feel that the theory of marginal productivity is a bit naive, I think for this top part of the labor market. That is to say when a manager manages to get a pay increase from $1 million a year to $10 million a year, according to the textbook based on marginal productivity, this should be due to the fact that his marginal contribution to the output of his company has risen from 1 to 10. Now it seems a bit naive. It could be that in practice individual marginal productivities are very hard to observe and monitor, especially in a large corporation. And there is clearly strong incentives for top managers to try to get as much as they can.

... Now, when the top tax rate is 82%, now of course you always want to be paid $1 million more, but on the margin when you get a pay increase of $1 million, 82% is going to go straight to the Treasury, so your incentive to bargain very aggressively and put the right people in the right compensation committee are going to be not so strong. And also your shareholders, your subordinates, maybe will tend to tell you, look, this is very costly. Whereas when the top tax rate goes down to 20, 30% or even 40%, so you keep 2/3rds or 60% of the extra $1 million for you, then the incentives are very, very different. Now, this model seems to explain part of what we observe in the data. In particular, it's very difficult to see any improvement in the performance of managers who are getting $10 million instead of $1 million. When we put together a data base with all the publicly traded companies in North America, Europe, Japan, trying to compare in the companies that are paying their managers $10 million instead of $1 million, it's very difficult to see in the data any extra performance.

... But let me make clear that I love capital accumulation and I certainly don't want to reduce capital accumulation. The problem is the concentration. So let me make very clear that inequality in itself is of course not a problem. Inequality can actually be useful for growth. Up to a point. The problem is when inequality of wealth and concentration of wealth gets too extreme, it is not useful any more for growth. And it can even become bad, because it leads to high perpetuation of inequality over time, so it can reduce social mobility. And it can also be bad for the working or for the democratic institutions. So where is the tipping point--when is it that inequality becomes excessive? Well, I'm sorry to tell you that I don't have a formula for that.

... In the United States right now, the bottom 50% of the population own about 2% of national wealth. And the next 40% own about 20, 22% of national wealth. And this group, the middle 40%, the people who are not in the bottom 50% and who are not in the top 10%, they used to own 25-30% of national wealth. And this has been going down in recent decades, as shown by a recent study by Saez and Zucman and now is closer to 20, 22%. Now, how much should it be? I don't know. I don't know. But the view that we need the middle class share to go down and down and down and that this is not a problem as long as you have positive growth, I think is excessive. You know, I think, of course we need entrepreneurs. I'm not saying, look, if it was perfect equality the bottom 50% should own 50% and the next 40% should own 40. I am not saying that we should have this at all. I'm just saying that when you have 2% for the bottom 50 and 22 for the next 40, you know, the view that we cannot do better than that [[ because ]] you won't have entrepreneurs any more, you won't have growth any more, is very ideological.

... I am actually a lot more optimistic than what some people seem to believe. I'm very sorry some people feel depressed after they read my book because after all this is not the way I wrote it. In fact, I think there are lots of reasons to be optimistic. For instance, one good news coming from the book is that we've never been as rich in terms of net wealth than we are today in developed countries. And we talk all the time about our public debt, but in fact our private wealth as a fraction of GDP has increased a lot more than our public debt as a fraction of GDP, so our national wealth, the sum of private and public wealth, is actually higher than it has ever been. So our countries are rich. It is our governments that are poor, which is a problem; but it raises issues of organization and institution but that can be addressed.

Ivy admissions discussed by parents of gifted kids

At this link you can find pages of discussion about Ivy admissions, stimulated at least in part by Pinker's recent article (see here and here), among parents of highly gifted children. Most of the discussants realize that there is a lot of room in the tail beyond the SAT ceiling. (Hint: parents of gifted kids tend to be fairly sharp themselves ... I wonder why? ;-)

I am not one of the commenters on the thread. Bonus points if you click the link above and read through to the Bezos quote ;-)
I suspect the 10 percent comment means something along the lines of 10 percent have been Intel semifinalists, have published significant research, qualified for USAMO, etc. That doesn't mean that the other 90 percent are dumb jocks and clueless legacies. The 90 percent probably includes some very bright, gifted kids, but they haven't cured cancer (not yet at least).
It means that 5%-10% are selected on academic merit alone. The rest are selected on a combination of factors. They may (mostly) have quite high academic merit, but other factors are considered, and so the overall academic merit of the class, though high, is less than it would have been if academic merit played a larger role in admissions. Students are admitted who are less academically meritorious than some who are rejected.


Regardless of what colleges supposedly should do, and what they do do, there is still the inescapable fact that SAT/ACT test have too low a ceiling, and the colleges are missing a huge amount of information about the academic ability of their applicants, and there is no excuse for them not actively pushing for harder tests.


They have a very good reason for refusing harder tests--it restricts their freedom. The first goal for colleges are self preservation and growth, hence the preference for legacies and athletics, both of which fuel alumni donations. But once self-preservation and growth has been achieved, college believe themselves to be forces for social engineering, helping right what they see as wrong in society.


Maybe it isn't about a harder test. Let's say that current perfect scores get you a group at a top school with IQs of 135+. Maybe it is 140+. With a different test, do you get 150+. But do you get a group that you want? Do they have the social skills to have a good mix, good clubs? There are factors that you want to have a certain type of school whether you are Harvard or Penn State. Harvard doesn't want a whole school that could pursue graduate work in Physics. They want fencing teams and rowing and a football team to play Yale. So for those of you wishing for a harder test, what does that mean to the student body, the college experience if you don't take into account all the other things. Because how much does it change if your roommate has an IQ of 175 in math, but 125 in ELA or 145 overall? I can see MIT wanting the 175 in math but Ivy's? Do you really want your kid going to a school where they just sit and have deep discussions about theories with other students?


But that assumes that those people are only interested in their peculiar "pointy" things. And that PG people lack social skills. Which is where my A versus B archetypes came from to begin with. Assume that they are BOTH HG+.

HG+ people come in a lot of different varieties there.

Just because someone has a FSIQ of 150+ doesn't mean that s/he is necessarily passionate about particle physics. It might mean that s/he is capable of learning it, but even that probably depends on the individual.

[[ IIUC, PG = Profoundly Gifted ; HG = Highly Gifted ]]

Do you really want your kid going to a school where they just sit and have deep discussions about theories with other students?
Where they just talk about big ideas? No. But I suspect that no one talks about nothing but big ideas, so the question is exaggerated.

As for a place where talking about big ideas is a normal part of the culture, yes, absolutely. Isn't that supposed to be the point about being at a place that calls itself a top-tier university --- that the people there are very bright and interested in big ideas in science, philosophy, history, and so on?

Saturday, September 20, 2014

How to build the future

I just bought 10 copies for my team at MSU. More here. The book is based on Thiel's Stanford class CS 183: Startup -- see course notes.

Earlier posts on Thiel.

Thursday, September 18, 2014

Excellent Sheep and Chinese Americans

Two recent podcasts I recommend. I disagree with Deresiewicz on many points (see my comments on Steve Pinker's response here and here), but the discussion is worth a listen.
Do the Best Colleges Produce the Worst Students?

As schools shift focus from the humanities to "practical" subjects like economics and computer science, students are losing the ability to think in innovative ways, argues William Deresiewicz. When he was a professor at Yale he noticed that his students, some of the nation’s brightest minds, seemed to be adrift when it came to knowing how to think critically and creatively and how to find a sense of purpose in life. Deresiewicz explains why he thinks college should be a time for self-discovery, when students can establish their own values and measures of success, so they can forge their own path. His book Excellent Sheep : The Miseducation of the American Elite and the Way to a Meaningful Life addresses parents, students, educators, and anyone who's interested in the direction of American society, exposing where the system is broken and presenting solutions.

Chinese Americans and the American Dream

In many ways, Chinese Americans today are exemplars of the American Dream—moving from indentured servitude to second-class status and outright exclusion to economic to social integration and achievement. But this narrative leaves a lot out. Eric Liu, author, educator, and entrepreneur, pieces together a sense of the Chinese American identity and looks at what it means to be Chinese American in this moment. His new book A Chinaman's Chance: One Family's Journey and the Chinese American Dream is a collection of personal essays that range from the meaning of Confucius to the role of Chinese Americans in shaping how we read the Constitution to why he hates the hyphen in "Chinese-American."

Sunday, September 14, 2014

Harvard admissions and meritocracy

Motivated by Steve Pinker's recent article The Trouble With Harvard (see my comments here), Ephblog drills down on Harvard admissions. The question is just how far Harvard deviates from Pinker's ideal of selecting the entire class based on intellectual ability. Others raised similar questions, as evidenced by, e.g., the very first comment that appeared on The New Republic's site:
JakeH 10 days ago

Great article. One quibble: Pinker says, based on "common knowledge," that only ten (or five) percent of Harvard students are selected based on academic merit, and that the rest are selected "holistically." His implication is that "holistic" consideration excludes academic merit as a major factor. But that's surely not the case. Even if Harvard only selects ten percent of its students based on academic factors alone, it seems likely that academic and test score standards are high for the remaining 90 percent. We don't have enough information on this point, because, I suppose, it's not available. (To solve that problem, I join Pinker's call for a more transparent admissions process.)
I don't know exactly how Harvard admissions works -- there are all sorts of mysteries. But let me offer the following observations.

1. Pinker claimed that only 5-10 percent of the class is admitted purely on the basis of academic merit (see more below). The 5-10 percent number was widely reported in the past, including by scholar Jerome Karabel. No one knows what Harvard is up to at the moment and it's possible that, given the high demand for elite education, they have increased their academic focus over the years.

2. IIRC, the current SAT ceiling of 1600 (M+CR) corresponds to about 1 in 1000 ability (someone please tell me if I am mistaken). So there are at least a couple thousand US kids per cohort at this ability level, and several times more who are near it ("within the noise"). A good admissions committee would look at other higher ceiling measures of ability (e.g., performance in math and science competitions) to rank order top applicants. The 800 ceiling on the math is not impressive at all -- a kid who is significantly below this level has almost no chance of mastering the Caltech required curriculum (hence even the 25th percentile math SAT score at Caltech is 770; in my day the attrition rate at Caltech was pretty high -- a lot of people "flamed out"). The reduced SAT ceiling makes it easier for Harvard to hide what it is up to.

4. My guess is that Harvard still has a category, in the past called S ("Scholar"; traditionally 5-10 percent of the class, but perhaps larger now), for the top rank-ordered candidates in academic ability alone. Most of the near-perfect scorers on the SAT will not qualify for S -- it is more impressive to have been a finalist in the Intel science competition, written some widely used/acclaimed code, made (or nearly made) the US IMO or IPhO teams, published some novel research or writing, etc. Harvard sometimes boasts about the number of perfect SAT scorers it rejects each year, so clearly one can't conclude that a 1600 on CR+M alone qualifies for the S category. Along these lines, one even reads occasional stories about Harvard rejecting IMO participants.

5. In remaining categories Harvard almost certainly uses a more holistic approach that also weights athletics, extracurriculars, etc. Some of the people who score high on this weighted measure might not have qualified in S, but nevertheless are near the ceiling in SAT score. It has been reported in the past that Harvard used a 1-5 scoring system in academics, sports, leadership, music, etc. and that to have serious consideration (outside the S category, which is for real superstars), one needed to have two or more "1" scores -- e.g., valedictorian/high SATs + state-level tennis player + ...

From the comments above, it should be clear that one can't simply use the percentage of near-perfect SAT scorers in the class to determine the size of the S category.

See here for discussion of meritocratic test-based systems in other countries. For instance, the Indian IIT, the French Ecole Normale Superieure, and the Taiwan university entrance exams, have in the past explicitly ranked the top scorers each year. (The tests are hard enough that typically no one gets near a perfect score; note things may have changed recently.) I know more than a few theoretical physicists who scored in the top 5 in their entire country on these exams. Mandlebrot writes in his autobiography about receiving the highest ENS score in France.

Friday, September 12, 2014

Embrace the Grind

Talent, hard work, and success in jiujitsu. "Show up every day and keep pushing through."

Wednesday, September 10, 2014

What is best for Harvard

I highly recommend Steve Pinker's The Trouble With Harvard in The New Republic.
... Like many observers of American universities, I used to believe the following story. Once upon a time Harvard was a finishing school for the plutocracy, where preppies and Kennedy scions earned gentleman’s Cs while playing football, singing in choral groups, and male-bonding at final clubs, while the blackballed Jews at CCNY founded left-wing magazines and slogged away in labs that prepared them for their Nobel prizes in science. Then came Sputnik, the '60s, and the decline of genteel racism and anti-Semitism, and Harvard had to retool itself as a meritocracy, whose best-and-brightest gifts to America would include recombinant DNA, Wall Street quants, The Simpsons, Facebook, and the masthead of The New Republic.

This story has a grain of truth in it: Hoxby has documented that the academic standards for admission to elite universities have risen over the decades. But entrenched cultures die hard, and the ghost of Oliver Barrett IV still haunts every segment of the Harvard pipeline.

At the admissions end, it’s common knowledge that Harvard selects at most 10 percent (some say 5 percent) of its students on the basis of academic merit. At an orientation session for new faculty, we were told that Harvard “wants to train the future leaders of the world, not the future academics of the world,” and that “We want to read about our student in Newsweek 20 years hence” (prompting the woman next to me to mutter, “Like the Unabomer”). The rest are selected “holistically,” based also on participation in athletics, the arts, charity, activism, travel, and, we inferred (Not in front of the children!), race, donations, and legacy status (since anything can be hidden behind the holistic fig leaf).

The lucky students who squeeze through this murky bottleneck find themselves in an institution that is single-mindedly and expensively dedicated to the pursuit of knowledge. It has an astonishing library system that pays through the nose for rare manuscripts, obscure tomes, and extortionately priced journals; exotic laboratories at the frontiers of neuroscience, regenerative medicine, cosmology, and other thrilling pursuits; and a professoriate with erudition in an astonishing range of topics, including many celebrity teachers and academic rock stars. The benefits of matching this intellectual empyrean with the world’s smartest students are obvious. So why should an ability to play the bassoon or chuck a lacrosse ball be given any weight in the selection process?

The answer, ironically enough, makes the admissocrats and Deresiewicz strange bedfellows: the fear of selecting a class of zombies, sheep, and grinds. But as with much in the Ivies’ admission policies, little thought has given to the consequences of acting on this assumption. Jerome Karabel has unearthed a damning paper trail showing that in the first half of the twentieth century, holistic admissions were explicitly engineered to cap the number of Jewish students. Ron Unz, in an exposé even more scathing than Deresiewicz’s, has assembled impressive circumstantial evidence that the same thing is happening today with Asians.

Just as troublingly, why are elite universities, of all institutions, perpetuating the destructive stereotype that smart people are one-dimensional dweebs? It would be an occasion for hilarity if anyone suggested that Harvard pick its graduate students, faculty, or president for their prowess in athletics or music, yet these people are certainly no shallower than our undergraduates. In any case, the stereotype is provably false. Camilla Benbow and David Lubinski have tracked a large sample of precocious teenagers identified solely by high performance on the SAT, and found that when they grew up, they not only excelled in academia, technology, medicine, and business, but won outsize recognition for their novels, plays, poems, paintings, sculptures, and productions in dance, music, and theater. A comparison to a Harvard freshman class would be like a match between the Harlem Globetrotters and the Washington Generals.
Pinker's position is similar to that of some of his distinguished predecessors on the Harvard faculty (see below). The ideal school Pinker is describing is called "Caltech" :-)
Defining Merit:
[The Chosen, Jerome Karabel] ... In a pair of letters that constituted something of a manifesto for the wing of the faculty favoring strict academic meritocracy, Wilson explicitly advocated admitting fewer private school students and commuters, eliminating all preferences for athletes, and (if funds permitted) selecting "the entering class regardless of financial need on the basis of pure merit." The issue of athletes particularly vexed Wilson, who stated flatly: "I would certainly rule out athletic ability as a criterion for admission of any sort," adding that "it bears a zero relationship to the performance later in life that we are trying to predict." He also argued that "it may well be that objective test scores are our only safeguards against an excessive number of athletes only, rich playboys, smooth characters who make a good impression in interviews, etc." As a parting shot, Wilson could not resist accusing Ford of anti-intellectualism; citing Ford's desire to change Harvard's image, Wilson asked bluntly: "What's wrong with Harvard being regarded as an egghead college? Isn't it right that a country the size of the United States should be able to afford one university in which intellectual achievement is the most important consideration?"
E. Bright Wilson was professor of chemistry and member of the National Academy of Sciences, later a recipient of the National Medal of Science. The last quote from Wilson could easily have come from anyone who went to Caltech! Indeed, both E. Bright Wilson and his son, Nobel Laureate Ken Wilson (theoretical physics), earned their doctorates at Caltech (the father under Linus Pauling, the son under Murray Gell-Mann). ...
Some have quibbled with Pinker's assertion that only 5 or 10% of the Harvard class is chosen with academic merit as the sole criterion. They note the overall high scores of Harvard students as evidence against this claim. But a simple calculation makes it obvious that the top 2000 or so high school seniors (including international students, who would eagerly attend Harvard if given the opportunity), ranked by brainpower alone, would be much stronger intellectually than the typical student admitted to Harvard today. (Vanderbilt researchers David Lubinski and Camilla Benbow, mentioned above by Pinker, study a population that is roughly 1 in 10k in ability. About two hundred US high school seniors with this level of talent are available each year; adding in international students increases the total significantly.)
Defining Merit: ... Bender also had a startlingly accurate sense of how many truly intellectually outstanding students were available in the national pool. He doubted whether more than 100-200 candidates of truly exceptional promise would be available for each year's class. This number corresponds to (roughly) +4 SD in mental ability. Long after Bender resigned, Harvard still reserved only 10 percent of its places (roughly 150 spots) for "top brains". (See category "S" listed at bottom.) ...

Typology used for all applicants, at least as late as 1988:

1. S First-rate scholar in Harvard departmental terms.
In the end, however, I have to agree with old Wilbur Bender, the Harvard admissions dean who fought off idealistic faculty committees in the the 1950s. A Harvard that followed Pinker's advice would, after a generation or two, be reduced in status, prestige, and endowment size, to a mere Caltech or Ecole Normale Superieure. (Both schools, by some estimates, produce Nobel Prize winning alumni at a rate several times higher than Harvard.)

What is good for our nation, and for civilization as a whole, is not what is best for Harvard.

Monday, September 08, 2014

Common genetic variants associated with cognitive performance

This is a follow up to earlier papers by the SSGAC collaboration -- see First GWAS Hits For Cognitive Ability and SNPs and SATS. Effect sizes found are typically ~ 0.3 IQ points. Someone with 50 more good variants (similar to these) than the average person would be about 1 SD above average in IQ.

Note among the authors names like Pinker, Visscher, Plomin, McGue, Deary, etc. Thank god it wasn't the sinister Chinese who got there first! For more on this topic, including the status of the BGI study, see Genetic Architecture of Intelligence (arXiv:1408.3421).
Common genetic variants associated with cognitive performance identified using the proxy-phenotype method (PNAS, doi: 10.1073/pnas.1404623111)

We identify common genetic variants associated with cognitive performance using a two-stage approach, which we call the proxy-phenotype method. First, we conduct a genome-wide association study of educational attainment in a large sample (n = 106,736), which produces a set of 69 education-associated SNPs. Second, using independent samples (n = 24,189), we measure the association of these education-associated SNPs with cognitive performance. Three SNPs (rs1487441, rs7923609, and rs2721173) are significantly associated with cognitive performance after correction for multiple hypothesis testing. In an independent sample of older Americans (n = 8,652), we also show that a polygenic score derived from the education-associated SNPs is associated with memory and absence of dementia. Convergent evidence from a set of bioinformatics analyses implicates four specific genes (KNCMA1, NRXN1, POU2F3, and SCRT). All of these genes are associated with a particular neurotransmitter pathway involved in synaptic plasticity, the main cellular mechanism for learning and memory.

Saturday, September 06, 2014

GameDay: Spartans vs Ducks

The Spartans hung in there (amazing that they were leading at the half) but ultimately the Ducks were too much.

I started out in the Spartan section but I went up to the Oregon President's box at halftime.

My t-shirt says "Spartan For Life" -- didn't go over very well with the Oregon fans! I took abuse everywhere except in the elitist box.

This is very Eugene: a human traffic jam crossing the footbridge over the Willamette river.

Friday, September 05, 2014

Back in Eugene

New UO science building:

Setting up for ESPN College GameDay:

Across the river to the stadium:

Ducks and Spartans play tomorrow:

I'm wrestling with divided loyalties. I've been a big Marcus Mariota fan since he came to UO, and I'd like to see him win a Heisman and a national title.

Craft brewing mecca:

@ Amazon

I was at the Amazon campus in Seattle yesterday to give a talk. Unbelievable construction and development in this part of town.

This is the famous Bezos "door desk" :-)

Wednesday, September 03, 2014


Did you miss out on the next big thing? :-)   See also Bitcoin dynamics.

Sunday, August 31, 2014

Metabolic costs of human brain development

This paper quantifies the unusually high energetic cost of brain development in humans. Brain energy requirements and body-weight growth rate are anti-correlated in childhood. Given these results it would be surprising if nutritional limitations that prevented individuals from achieving their genetic potential in height didn't also lead to sub-optimal cognitive development. Nutritional deprivation likely stunts both mind and body.

See also Brainpower ain't free.
Metabolic costs and evolutionary implications of human brain development
(PNAS doi:10.1073/pnas.1323099111)

The metabolic costs of brain development are thought to explain the evolution of humans’ exceptionally slow and protracted childhood growth; however, the costs of the human brain during development are unknown. We used existing PET and MRI data to calculate brain glucose use from birth to adulthood. We find that the brain’s metabolic requirements peak in childhood, when it uses glucose at a rate equivalent to 66% of the body’s resting metabolism and 43% of the body’s daily energy requirement, and that brain glucose demand relates inversely to body growth from infancy to puberty. Our findings support the hypothesis that the unusually high costs of human brain development require a compensatory slowing of childhood body growth.

The high energetic costs of human brain development have been hypothesized to explain distinctive human traits, including exceptionally slow and protracted preadult growth. Although widely assumed to constrain life-history evolution, the metabolic requirements of the growing human brain are unknown. We combined previously collected PET and MRI data to calculate the human brain’s glucose use from birth to adulthood, which we compare with body growth rate. We evaluate the strength of brain–body metabolic trade-offs using the ratios of brain glucose uptake to the body’s resting metabolic rate (RMR) and daily energy requirements (DER) expressed in glucose-gram equivalents (glucosermr% and glucoseder%). We find that glucosermr% and glucoseder% do not peak at birth (52.5% and 59.8% of RMR, or 35.4% and 38.7% of DER, for males and females, respectively), when relative brain size is largest, but rather in childhood (66.3% and 65.0% of RMR and 43.3% and 43.8% of DER). Body-weight growth (dw/dt) and both glucosermr% and glucoseder% are strongly, inversely related: soon after birth, increases in brain glucose demand are accompanied by proportionate decreases in dw/dt. Ages of peak brain glucose demand and lowest dw/dt co-occur and subsequent developmental declines in brain metabolism are matched by proportionate increases in dw/dt until puberty. The finding that human brain glucose demands peak during childhood, and evidence that brain metabolism and body growth rate covary inversely across development, support the hypothesis that the high costs of human brain development require compensatory slowing of body growth rate.

Thursday, August 28, 2014

Determination of Nonlinear Genetic Architecture using Compressed Sensing

It is a common belief in genomics that nonlinear interactions (epistasis) in complex traits make the task of reconstructing genetic models extremely difficult, if not impossible. In fact, it is often suggested that overcoming nonlinearity will require much larger data sets and significantly more computing power. Our results show that in broad classes of plausibly realistic models, this is not the case.
Determination of Nonlinear Genetic Architecture using Compressed Sensing (arXiv:1408.6583)
Chiu Man Ho, Stephen D.H. Hsu
Subjects: Genomics (q-bio.GN); Applications (stat.AP)

We introduce a statistical method that can reconstruct nonlinear genetic models (i.e., including epistasis, or gene-gene interactions) from phenotype-genotype (GWAS) data. The computational and data resource requirements are similar to those necessary for reconstruction of linear genetic models (or identification of gene-trait associations), assuming a condition of generalized sparsity, which limits the total number of gene-gene interactions. An example of a sparse nonlinear model is one in which a typical locus interacts with several or even many others, but only a small subset of all possible interactions exist. It seems plausible that most genetic architectures fall in this category. Our method uses a generalization of compressed sensing (L1-penalized regression) applied to nonlinear functions of the sensing matrix. We give theoretical arguments suggesting that the method is nearly optimal in performance, and demonstrate its effectiveness on broad classes of nonlinear genetic models using both real and simulated human genomes.
Click for larger image.

Cosmopolitans -- Whit Stillman returns on Amazon

The pilot isn't bad -- American expats in Paris :-) The cinematography is beautiful, but then it's hard to go wrong in Paris.

More Whit Stillman.

Rabbit genome: domestication via soft sweeps

Domestication -- genetic change in response to a drastic change in environment -- happened via allele frequency changes at many loci. I expect a similar pattern in humans due to, e.g., agriculture.

I don't know why some researchers find this result surprising -- it seemed quite likely to me that "adaptation to domestication" is a complex trait controlled by many loci. Hence a shift in the phenotype is likely to be accomplished through frequency changes at many alleles.
Rabbit genome analysis reveals a polygenic basis for phenotypic change during domestication (Science DOI: 10.1126/science.1253714)

The genetic changes underlying the initial steps of animal domestication are still poorly understood. We generated a high-quality reference genome for the rabbit and compared it to resequencing data from populations of wild and domestic rabbits. We identified more than 100 selective sweeps specific to domestic rabbits but only a relatively small number of fixed (or nearly fixed) single-nucleotide polymorphisms (SNPs) for derived alleles. SNPs with marked allele frequency differences between wild and domestic rabbits were enriched for conserved noncoding sites. Enrichment analyses suggest that genes affecting brain and neuronal development have often been targeted during domestication. We propose that because of a truly complex genetic background, tame behavior in rabbits and other domestic animals evolved by shifts in allele frequencies at many loci, rather than by critical changes at only a few domestication loci.
From the paper:
... directional selection events associated with rabbit domestication are consistent with polygenic and soft sweep modes of selection (18) that primarily acted on standing genetic variation in regulatory regions of the genome. This stands in contrast with breed-specific traits in many domesticated animals that often show a simple genetic basis with complete fixation of causative alleles (19). Our finding that many genes affecting brain and neuronal development have been targeted during rabbit domestication is fully consistent with the view that the most critical phenotypic changes during the initial steps of animal domestication probably involved behavioral traits that allowed animals to tolerate humans and the environment humans offered. On the basis of these observations, we propose that the reason for the paucity of specific fixed domestication genes in animals is that no single genetic change is either necessary or sufficient for domestication. Because of the complex genetic background for tame behavior, we propose that domestic animals evolved by means of many mutations of small effect, rather than by critical changes at only a few domestication loci.
I'll repeat again that simply changing a few hundred allele frequencies in humans could make us much much smarter ...

Wednesday, August 27, 2014

Neural Networks and Deep Learning 2

Inspired by the topics discussed in this earlier post, I've been reading Michael Nielsen's online book on neural nets and deep learning. I particularly liked the subsection quoted below. For people who think deep learning is anything close to a solved problem, or anticipate a near term, quick take-off to the Singularity, I suggest they read the passage below and grok it deeply.
Neural Networks and Deep Learning (Chapter 3):

You have to realize that our theoretical tools are very weak. Sometimes, we have good mathematical intuitions for why a particular technique should work. Sometimes our intuition ends up being wrong [...] The questions become: how well does my method work on this particular problem, and how large is the set of problems on which it works well. -- Question and answer with neural networks researcher Yann LeCun

Once, attending a conference on the foundations of quantum mechanics, I noticed what seemed to me a most curious verbal habit: when talks finished, questions from the audience often began with "I'm very sympathetic to your point of view, but [...]". Quantum foundations was not my usual field, and I noticed this style of questioning because at other scientific conferences I'd rarely or never heard a questioner express their sympathy for the point of view of the speaker. At the time, I thought the prevalence of the question suggested that little genuine progress was being made in quantum foundations, and people were merely spinning their wheels. Later, I realized that assessment was too harsh. The speakers were wrestling with some of the hardest problems human minds have ever confronted. Of course progress was slow! But there was still value in hearing updates on how people were thinking, even if they didn't always have unarguable new progress to report.

You may have noticed a verbal tic similar to "I'm very sympathetic [...]" in the current book. To explain what we're seeing I've often fallen back on saying "Heuristically, [...]", or "Roughly speaking, [...]", following up with a story to explain some phenomenon or other. These stories are plausible, but the empirical evidence I've presented has often been pretty thin. If you look through the research literature you'll see that stories in a similar style appear in many research papers on neural nets, often with thin supporting evidence. What should we think about such stories?

In many parts of science - especially those parts that deal with simple phenomena - it's possible to obtain very solid, very reliable evidence for quite general hypotheses. But in neural networks there are large numbers of parameters and hyper-parameters, and extremely complex interactions between them. In such extraordinarily complex systems it's exceedingly difficult to establish reliable general statements. Understanding neural networks in their full generality is a problem that, like quantum foundations, tests the limits of the human mind. Instead, we often make do with evidence for or against a few specific instances of a general statement. As a result those statements sometimes later need to be modified or abandoned, when new evidence comes to light.

[ Sufficiently advanced AI will come to resemble biology, even psychology, in its complexity and resistance to rigorous generalization ... ]

One way of viewing this situation is that any heuristic story about neural networks carries with it an implied challenge. For example, consider the statement I quoted earlier, explaining why dropout works* *From ImageNet Classification with Deep Convolutional Neural Networks by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton (2012).: "This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons." This is a rich, provocative statement, and one could build a fruitful research program entirely around unpacking the statement, figuring out what in it is true, what is false, what needs variation and refinement. Indeed, there is now a small industry of researchers who are investigating dropout (and many variations), trying to understand how it works, and what its limits are. And so it goes with many of the heuristics we've discussed. Each heuristic is not just a (potential) explanation, it's also a challenge to investigate and understand in more detail.

Of course, there is not time for any single person to investigate all these heuristic explanations in depth. It's going to take decades (or longer) for the community of neural networks researchers to develop a really powerful, evidence-based theory of how neural networks learn. Does this mean you should reject heuristic explanations as unrigorous, and not sufficiently evidence-based? No! In fact, we need such heuristics to inspire and guide our thinking. It's like the great age of exploration: the early explorers sometimes explored (and made new discoveries) on the basis of beliefs which were wrong in important ways. Later, those mistakes were corrected as we filled in our knowledge of geography. When you understand something poorly - as the explorers understood geography, and as we understand neural nets today - it's more important to explore boldly than it is to be rigorously correct in every step of your thinking. And so you should view these stories as a useful guide to how to think about neural nets, while retaining a healthy awareness of the limitations of such stories, and carefully keeping track of just how strong the evidence is for any given line of reasoning. Put another way, we need good stories to help motivate and inspire us, and rigorous in-depth investigation in order to uncover the real facts of the matter.
See also here from an earlier post:
... evolution has [ encoded the results of a huge environment-dependent optimization ] in the structure of our brains (and genes), a process that AI would have to somehow replicate. A very crude estimate of the amount of computational power used by nature in this process leads to a pessimistic prognosis for AI even if one is willing to extrapolate Moore's Law well into the future. [ Moore's Law (Dennard scaling) may be toast for the next decade or so! ] Most naive analyses of AI and computational power only ask what is required to simulate a human brain, but do not ask what is required to evolve one. I would guess that our best hope is to cheat by using what nature has already given us -- emulating the human brain as much as possible.
If indeed there are good (deep) generalized learning architectures to be discovered, that will take time. Even with such a learning architecture at hand, training it will require interaction with a rich exterior world -- either the real world (via sensors and appendages capable of manipulation) or a computationally expensive virtual world. Either way, I feel confident in my bet that a strong version of the Turing test (allowing, e.g., me to communicate with the counterpart over weeks or months; to try to teach it things like physics and watch its progress; eventually for it to teach me) won't be passed until at least 2050 and probably well beyond.

Turing as polymath: ... In a similar way Turing found a home in Cambridge mathematical culture, yet did not belong entirely to it. The division between 'pure' and 'applied' mathematics was at Cambridge then as now very strong, but Turing ignored it, and he never showed mathematical parochialism. If anything, it was the attitude of a Russell that he acquired, assuming that mastery of so difficult a subject granted the right to invade others.

Friday, August 22, 2014

Two reflections on SCI FOO 2014

Two excellent blog posts on SCI FOO by Jacob Vanderplas (Astronomer and Data Scientist at the University of Washington) and Dominic Cummings (former director of strategy for the conservative party in the UK).

Hacking Academia: Data Science and the University (Vanderplas)

Almost a year ago, I wrote a post I called the Big Data Brain Drain, lamenting the ways that academia is neglecting the skills of modern data-intensive research, and in doing so is driving away many of the men and women who are perhaps best equipped to enable progress in these fields. This seemed to strike a chord with a wide range of people, and has led me to some incredible opportunities for conversation and collaboration on the subject. One of those conversations took place at the recent SciFoo conference, and this article is my way of recording some reflections on that conversation. ...

The problem we discussed is laid out in some detail in my Brain Drain post, but a quick summary is this: scientific research in many disciplines is becoming more and more dependent on the careful analysis of large datasets. This analysis requires a skill-set as broad as it is deep: scientists must be experts not only in their own domain, but in statistics, computing, algorithm building, and software design as well. Many researchers are working hard to attain these skills; the problem is that academia's reward structure is not well-poised to reward the value of this type of work. In short, time spent developing high-quality reusable software tools translates to less time writing and publishing, which under the current system translates to little hope for academic career advancement. ...

Few scientists know how to use the political system to effect change. We need help from people like Cummings.

... It was interesting that some very eminent scientists, all much cleverer than ~100% of those in politics [INSERT: better to say 'all with higher IQ than ~100% of those in politics'], have naive views about how politics works. In group discussions, there was little focused discussion about how they could influence politics better even though it is clearly a subject that they care about very much. (Gershenfeld said that scientists have recently launched a bid to take over various local government functions in Barcelona, which sounds interesting.)

... To get things changed in politics, scientists need mechanisms a) to agree priorities in order to focus their actions on b) roadmaps with specifics. Generalised whining never works. The way to influence politicians is to make it easy for them to fall down certain paths without much thought, and this means having a general set of goals but also a detailed roadmap the politicians can apply, otherwise they will drift by default to the daily fog of chaos and moonlight.


3. High status people have more confidence in asking basic / fundamental / possibly stupid questions. One can see people thinking ‘I thought that but didn’t say it in case people thought it was stupid and now the famous guy’s said it and everyone thinks he’s profound’. The famous guys don’t worry about looking stupid and they want to get down to fundamentals in fields outside their own.

4. I do not mean this critically but watching some of the participants I was reminded of Freeman Dyson’s comment:

‘I feel it myself, the glitter of nuclear weapons. It is irresistible if you come to them as a scientist. To feel it’s there in your hands. To release the energy that fuels the stars. To let it do your bidding. And to perform these miracles, to lift a million tons of rock into the sky, it is something that gives people an illusion of illimitable power, and it is in some ways responsible for all our troubles... this is what you might call ‘technical arrogance’ that overcomes people when they see what they can do with their minds.’

People talk about rationales for all sorts of things but looking in their eyes the fundamental driver seems to be – am I right, can I do it, do the patterns in my mind reflect something real? People like this are going to do new things if they can and they are cleverer than the regulators. As a community I think it is fair to say that outside odd fields like nuclear weapons research (which is odd because it still requires not only a large collection of highly skilled people but also a lot of money and all sorts of elements that are hard (but not impossible) for a non-state actor to acquire and use without detection), they believe that pushing the barriers of knowledge is right and inevitable. ...

Sunday, August 17, 2014

Genetic Architecture of Intelligence (arXiv:1408.3421)

This paper is based on talks I've given in the last few years. See here and here for video. Although there isn't much that hasn't already appeared in the talks or on this blog (other than some Compressed Sensing results for the nonlinear case) it's nice to have it in one place. The references are meant to be useful to people seriously interested in this subject, although I imagine they are nowhere near comprehensive. Apologies to anyone whose work I missed.

If you don't like the word "intelligence" just substitute "height" and everything will be OK. We live in strange times.
On the genetic architecture of intelligence and other quantitative traits (arXiv:1408.3421)
Categories: q-bio.GN
Comments: 30 pages, 13 figures

How do genes affect cognitive ability or other human quantitative traits such as height or disease risk? Progress on this challenging question is likely to be significant in the near future. I begin with a brief review of psychometric measurements of intelligence, introducing the idea of a "general factor" or g score. The main results concern the stability, validity (predictive power), and heritability of adult g. The largest component of genetic variance for both height and intelligence is additive (linear), leading to important simplifications in predictive modeling and statistical estimation. Due mainly to the rapidly decreasing cost of genotyping, it is possible that within the coming decade researchers will identify loci which account for a significant fraction of total g variation. In the case of height analogous efforts are well under way. I describe some unpublished results concerning the genetic architecture of height and cognitive ability, which suggest that roughly 10k moderately rare causal variants of mostly negative effect are responsible for normal population variation. Using results from Compressed Sensing (L1-penalized regression), I estimate the statistical power required to characterize both linear and nonlinear models for quantitative traits. The main unknown parameter s (sparsity) is the number of loci which account for the bulk of the genetic variation. The required sample size is of order 100s, or roughly a million in the case of cognitive ability.

Saturday, August 16, 2014

Neural Networks and Deep Learning

One of the SCI FOO sessions I enjoyed the most this year was a discussion of deep learning by AI researcher Juergen Schmidhuber. For an overview of recent progress, see this recent paper. Also of interest: Michael Nielsen's pedagogical book project.

An application which especially caught my attention is described by Schmidhuber here:
Many traditional methods of Evolutionary Computation [15-19] can evolve problem solvers with hundreds of parameters, but not millions. Ours can [1,2], by greatly reducing the search space through evolving compact, compressed descriptions [3-8] of huge solvers. For example, a Recurrent Neural Network [34-36] with over a million synapses or weights learned (without a teacher) to drive a simulated car based on a high-dimensional video-like visual input stream.
More details here. They trained a deep neural net to drive a car using visual input (pixels from the driver's perspective, generated by a video game); output consists of steering orientation and accelerator/brake activation. There was no hard coded structure corresponding to physics -- the neural net optimized a utility function primarily defined by time between crashes. It learned how to drive the car around the track after less than 10k training sessions.

For some earlier discussion of deep neural nets and their application to language translation, see here. Schmidhuber has also worked on Solomonoff universal induction.

These TED videos give you some flavor of Schmidhuber's sense of humor :-) Apparently his younger brother (mentioned in the first video) has transitioned from theoretical physics to algorithmic finance. Schmidhuber on China.

Friday, August 15, 2014

Y Combinator: "fund for the pivot"

I'm catching up on podcasts a bit now that I'm back in Michigan. I had an iTunes problem and was waiting for the next version release while on the road.

Econtalk did a nice interview with Y Combinator President Sam Altman. Y Combinator has always been entrepreneur-centric, to the point that the quality of the founders is one of the main factors they consider (i.e., more important than startup idea or business plan). At around 19 minutes, Altman reveals that they often "fund for the pivot" -- meaning that sometimes they want to place a bet on the entrepreneur even if they think the original idea is doomed. Altman also reveals that Y Combinator never looks at business plans or revenue projections. I can't count the number of times an idiot MBA demanded a detailed revenue projection from one of my startups, at a stage where the numbers and projections were completely meaningless.

Another good observation is about the importance of communication skills in a founder. The leadership team are a central nexus that has to informationally equilibrate the rest of the company + investors + partners + board members + journalists + customers ...  This is benefited tremendously by having someone who is articulate, succinct, and can "code switch" so as to speak the native language of an engineer or sales rep or VC.

@30 min or so:
Russ: ... one of the things that happens to me when I come out here in the summer--I live outside of Washington, D.C. and I come out every 6 or 7 weeks in the summer, and come to Stanford--I feel like I'm at the center of the universe. You know, Washington is--everyone in Washington, except for me--

Guest: Thinks they are--

Russ: Thinks they are in the center. And there are things they are in the center in. Obviously. But it's so placid there. And when I come to Stanford, the intellectual, the excitement about products and transforming concepts into reality, is palpable. And then I run into start-up people and venture capitalists. And they are so alive, compared to, say, a lobbyist in Washington, say, just to pick a random example. And there are certain things that just--again, it's almost palpable. You can almost feel them. So the thing is that I notice being here--which are already the next big thing, which at least they feel like they are.  [ Visiting Washington DC gives me hives! ]
I recall a Foo Camp (the O'Reilly one, not SCI FOO at Google; perhaps 2007-2010 or so) session led by Paul Graham and some of the other Y Combinator founders/funders. At the time they weren't sure at all that their model would work. It was quite an honest discussion and I think even they must be surprised at how successful they've been since then.

Wednesday, August 13, 2014

Designer babies: selection vs editing

The discussion in this video is sophisticated enough to make the distinction between embryo selection -- the parents get a baby whose DNA originates from them, but the "best baby possible" -- and active genetic editing, which can give the child genes that neither parent had.

The movie GATTACA focuses on selection -- the director made a deliberate decision to eliminate reference to splicing or editing of genes. (Possibly because Ethan Hawke's character Vincent would have no chance competing against edited people.)

At SCI FOO, George Church seemed confident that editing would be an option in the near future. He is convinced that off-target mutations are not a problem for CRISPR. I have not yet seen this demonstrated in the literature, but of course George knows a lot more than what has been published. (Warning: I may have misunderstood his comments as there was a lot of background noise when we were talking.)

One interesting genetic variant (Lrp5?) that I learned about at the meeting, of obvious interest to future splicers and editors, apparently conveys an +8 SD increase in bone strength!

My views on all of this:
... given sufficient phenotype|genotype data, genomic prediction of traits such as cognitive ability will be possible. If, for example, 0.6 or 0.7 of total population variance is captured by the predictor, the accuracy will be roughly plus or minus half a standard deviation (e.g., a few cm of height, or 8 IQ points). The required sample size to extract a model of this accuracy is probably on the order of a million individuals. As genotyping costs continue to decline, it seems likely that we will reach this threshold within five years for easily acquired phenotypes like height (self-reported height is reasonably accurate), and perhaps within the next decade for more difficult phenotypes such as cognitive ability. At the time of this writing SNP genotyping costs are below $50 USD per individual, meaning that a single super-wealthy benefactor could independently fund a crash program for less than $100 million.

Once predictive models are available, they can be used in reproductive applications, rang- ing from embryo selection (choosing which IVF zygote to implant) to active genetic editing (e.g., using powerful new CRISPR techniques). In the former case, parents choosing between 10 or so zygotes could improve their expected phenotype value by a population standard de- viation. For typical parents, choosing the best out of 10 might mean the difference between a child who struggles in school, versus one who is able to complete a good college degree. Zygote genotyping from single cell extraction is already technically well developed [25], so the last remaining capability required for embryo selection is complex phenotype prediction. The cost of these procedures would be less than tuition at many private kindergartens, and of course the consequences will extend over a lifetime and beyond.

The corresponding ethical issues are complex and deserve serious attention in what may be a relatively short interval before these capabilities become a reality. Each society will decide for itself where to draw the line on human genetic engineering, but we can expect a diversity of perspectives. Almost certainly, some countries will allow genetic engineering, thereby opening the door for global elites who can afford to travel for access to reproductive technology. As with most technologies, the rich and powerful will be the first beneficiaries. Eventually, though, I believe many countries will not only legalize human genetic engineering, but even make it a (voluntary) part of their national healthcare systems [26]. The alternative would be inequality of a kind never before experienced in human history.

Here is the version of the GATTACA scene that was cut. The parents are offered the choice of edited or spliced genes conferring rare mathematical or musical ability.

Monday, August 11, 2014

SCI FOO 2014: photos

The day before SCI FOO I visited Complete Genomics, which is very close to the Googleplex.

Self-driving cars:

SCI FOO festivities:

I did an interview with O'Reilly. It should appear in podcast form at some point and I'll post a link.

Obligatory selfie:

Blog Archive