Replication


❝   And that’s why mistakes had to be corrected. BASF fully recognized that Ostwald would be annoyed by criticism of his work. But they couldn’t tiptoe around it, because they were trying to make ammonia from water and air. If Ostwald’s work couldn’t help them do that, then they couldn’t get into the fertilizer and explosives business. They couldn’t make bread from air. And they couldn’t pay Ostwald royalties. If the work wasn’t right, it was useless to everyone, including Ostwald.

From Paul von Hippel, "[When does science self-correct?](https://goodscience.substack.com/p/when-does-science-self-correct-lessons)". And when not.

A friend sent me 250th Anniversary Boston Tea Party tea for the 16th.

The tea must be authentic: it’s appears to have been intercepted in transit.

Intelligible Failure

Adam Russell created the DARPA SCORE replication project. Here he reflects on the importance of Intelligible Failure.

[Advanced Research Projects Agencies] need intelligible failure to learn from the bets they take. And that means evaluating risks taken (or not) and understanding—not merely observing—failures achieved, which requires both brains and guts. That brings me back to the hardest problem in making failure intelligible: ourselves. Perhaps the neologism we really need going forward is for intelligible failure itself—to distinguish it, as a virtue, from the kind of failure that we never want to celebrate: the unintelligible failure, immeasurable, born of sloppiness, carelessness, expediency, low standards, or incompetence, with no way to know how or even if it contributed to real progress.

Came across an older Alan Jacobs post:

For those who have been formed largely by the mythical core of human culture, disagreement and alternative points of view may well appear to them not as matters for rational adjudication but as defilement from which they must be cleansed.

It also has a section on The mythical core as lossy compression.

Today I listened to Sam Harris and read Alan Jacobs on Israel & Gaza. Highly recommended.

(With luck “Micropost” will link Jacobs here.)

Huh. Moonlight is redder than sunlight. The “silvery moon” is an illusion. iopscience.iop.org/article/1…

This looks like a good way to appreciate the beautiful bright thing that periodically makes it hard to see nebulas. blog.jatan.space

Do your own research

Sabine Hossenfelder’s Do your own research… but do it right is an excellent guide to critical thinking and a helpful antidote to the meme that no one should “do your own research”.

  1. (When not to do you own research.)
  2. Prepare well:
    • Reasonable expections: what can you reasonably learn in hours of online/library work?
    • Which specific questions are you trying to answer?
    • Be honest with yourself: about your biases and about what you don’t understand, or aren’t understanding as you read.
  3. Start with basics: Begin wtih peer-reviewed review articles, reports, lectures, & textbooks. Then look at recent publications. Use Google Scholar and related services to track citations to your source. Check for predatory journals. Beware preprints and conference proceedings, unless you can consult an expert.
  4. No cherry-picking! [ Even though you probably started because someone is wrong on the internet. -crt] This is the #1 mistake of “do your own research”.
  5. Track down sources
    • Never trust 2nd hand sources. Look at them to get started, but don’t end there.
    • If data is available, favor that over the text. Abstracts and conclusions especially tend to overstate.

Strong Towns & Ideological Purity

Good essay by Peter Norton Why we need Strong Towns critiquing a Current Affairs piece by Allison Dean.

I side with Norton here: Dean falls into the trap of demanding ideological purity. If you have to read only one, pick Norton. But after donning your Norton spectacles, read Dean for a solid discussion of points of overlap, reinterpreting critiques as debate about the best way to reach shared goals.

This other response to Dean attempts to add some middle ground to Dean’s Savannah and Flint examples, hinting what a reframed critique might look like. Unfortunately it’s long and meanders, and grinds its own axen.

I suspect Dean is allergic to economic justifications like “wealth” and “prosperity”. But we want our communities to thrive, and valuing prosperity is no more yearning for Dickensian hellscapes than loving community is pining for totalitarian ones.

The wonderful, walkable, wish-I-lived-there communities on Not Just Bikes are thriving, apparently in large part by sensible people-oriented design. More of that please.

(And watch Not Just Bikes for a more approachable take on Strong Towns, and examples of success.)

Essentialism

In a recent newsletter, Jesse Singal notes that recent MAGA gains among Democratic constituencies should prompt progressives to pause and question their own political assumptions & theories:

Quote from Singal saying Trump's continued inroads should prompt soul-searching from the Left: what have they got wrong?

It is quite a string of anomalies. A scientist would be prompted to look for alternate theories.

Singal suggests part of the problem is essentialism:

…activists and others like to talk and write about race in the deeply essentialist and condescending and tokenizing way… It’s everywhere, and it has absolutely exploded during the Trump years.

…both right-wing racists and left-of-center social justice types, [tend] to flatten groups of hundreds of millions of people into borderline useless categories, and to then pretend they share some sort of essence…

The irony.

~~~

Aside: I’m also reminded of a grad school story from Ruth.

Zeno: …Descartes is being an essentialist here…

Ruth: Wait, no, I think you’re being the essentialist….

_____: I’m sorry, what’s an essentialist?

Zeno: [short pause] It is a derogatory term.

(Yes, we had a philosophy teacher named Zeno. I’m not sure if the subject was actually Descartes.)

Unforced bias

In The bias we swim in, Linda McIver notes:

Recently I saw a post going around about how ChatGPT assumed that, in the sentence “The paralegal married the attorney because she was pregnant.” “she” had to refer to the paralegal. It went through multiple contortions to justify its assumption…

Her own conversation with ChatGPT was not as bad as the one making the rounds, but still self-contradictory.

Of course it makes the common gender mistake. What amazes me are the contorted justifications. What skin off AIs nose to say, “Yeah, OK, ‘she’ could be the attorney”? But it’s also read responses and learned that humans get embarrassed and move to self defense.

                         ~~~~~

If justifying, it could do better. Surely someone must have written how the framing highlights the power dynamic. And I find this reading less plausible:

The underling married his much better-paid boss BECAUSE she was pregnant.

At least, it’s not the same BECAUSE implied in the original.

Actual history (Cowen)

Reflection for sheltered people like me:


❝  

For my entire life, and a bit more, … virtually all of us [in the US] have been living in a bubble “outside of history.”

Hardly anyone you know, including yourself, is prepared to live in actual “moving” history. It will panic many of us, disorient the rest of us, and cause great upheavals in our fortunes, both good and bad. In my view, the good will considerably outweigh the bad (at least from losing #2 [absence of radical techological change], not #1 [American hegemony]), but I do understand that the absolute quantity of the bad disruptions will be high.

~Tyler Cowen, “There is no turning back on AI” Free Press Marginal Rev


Laura points out that even in the US, a sizable minority have been living in “actual history”.

UntitledImage

Alan Jacobs, The HedgeHog Review, “David Hume’s Guide to Social Media: Emancipation by the cultivation of taste.”

Alan Jacobs reminds us to eschew easy opinions (re: Supreme court here): blog.ayjay.org/reading-s…

…in which minutes are kept and hours are lost. 😉

Geol. Soc. of Wash. 1999

Twitter screenshot: came across this in the strongtown.org newsletter.

AstralCodex on nerds and hipsters:

Revisited my old post about reconstructing Syrotuck’s (lost) Lost Person Data. Mediocre writing, but I still like the basic data detective work we did.

Analysis in Google Sheets

Bookmark: Haidt on Why the mental health of liberal girls sank first and fastest.

  • Reverse CBT hypothesis
  • Tumblr - 4Chan dynamic (From Phelps-Roper)
  • Other things to re-read

Misunderstanding the replication crisis

Based on the abstract, it seems Alexander Bird’s Understanding the replication crisis as a base rate fallacy has it backwards. Is there reason to dig into the paper?

He notes a core feature of the crisis:

If most of the hypotheses under test are false, then there will be many false hypotheses that are apparently supported by the outcomes of well conducted experiments and null hypothesis significance tests with a type-I error rate (α) of 5%.

Then he says this solves the problem:

Failure to recognize this is to commit the fallacy of ignoring the base rate.

But it merely states the problem: Why most published research findings are false.

LOTR Musing

Fascinating discussion of Lord of the Rings:

But whatever it is, it seems to whisper of the sovereignty of mercy above that of legal decree. It shows us a world in which penalties of death are declared, but are then abrogated by the wise and kind.

And on this theme, Laura reminds me of the classic essay, Frodo didn’t fail

I think Jacobs has the right of it where he stakes a claim, but there’s much else in Roberts' pieces. Including this marvelous merger of Eowyn’s speech at Pelennor and the Gettysburg address.

It seemed that Dernhelm laughed, and the clear voice was like the ring of steel. ‘But no living man am I! You look upon a woman. Eowyn I am, Eomund’s daughter. You stand between me and my lord and kin. Begone, if you be not deathless! For living or dark undead, I will smite you, if you touch him. It is for us the living, rather, to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us — that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion — that we here highly resolve that these dead shall not have died in vain. That &c. &c.’

Roberts prefers the punchier movie version, “I am no man”. Fair enough! But in musing on that I found a fantastic 2015 essay by Mary Huening, '“I am no man” doesn’t cut it' that lays out precisely what we lose about Eowyn in the movies.

Alan Jacobs on The sovereignty of mercy in Middle Earth (and beyond).

I understand?

I enjoyed this discussion of the John Snow cholera story, inspired by Christmas cookies. podcasts.apple.com

To fix peer review, break it into stages, a short Nature opinion by Olava Amaral.

  • Separate evaluation of rigor from curation of space.
  • Check the data first!
  • Statcheck can do a lot of this.
  • Reallocate the 100M annual hours of peer review to standardization etc.