Checking in on the Covid19 Experts

About 5 weeks into a combination of thoughtfully reduced mitigation restrictions and riots, cases are rising in the USA. Deaths, in some places, are climbing but overall they continue to fall, from 2200/day at the peak in mid-April  to 650 today. Cases are not rising geometrically, as they did in late Mar and early Apr. Indeed, new cases declined slightly in May, and rose slightly in Jun. Alarmist headlines about record numbers of cases are correct, ignore the fact that new cases have been “largely flat” for about 3 months.

Case are likely rising due to a combination of dramatically increased testing, and genuine increases in infection rates due to relaxed mitigations.

Cases are a leading indicator, and deaths may follow.

It’s also possible that new cases are being found in the younger, healthier population, and that deaths will not track cases.

“Phased openings” are being followed in most states. The actual goals are unclear.

Per Washington State’s plan, each phase has entrance criteria, which track things like infection rates and hospital utilization, and a prescribed reduction of mitigations. People are allowed to gather in larger groups, sit closer together etc.

The plan seems to be that as the pandemic subsides, we gradually return to more risky behaviors.

Assuming that it was the mitigations that arrested the geometric rise, it is unclear why returning  to more risky behaviors will not restart it. If it wasn’t, perhaps we didn’t need them in the first place.

Perhaps the plan is to see-saw back on forth between phases for years.

Perhaps we have no idea what we are doing.

Posted in Uncategorized | Leave a comment

A Brief History of Corona19 in America

Late Feb 2019: This Might Be Serious

The fundamental change in America was an awakening from 10 years of unseriousness. At some level, everyone understands, and always has, that impeachment was nonsense, there are two genders, global warming is a political movement, and that infinite debt is not OK. But in Jan 2020 lots of people pretended otherwise. Life Care ended all that.

Mar 2019: The New Politics

$3,000,000,000,000 in new debt. Accusations that a lack of a uniformed response in all states would cause untold death. Travel bans are racist. 50,000 ventilators. This is all Trump’s fault, obviously. Millions could die.

Mar 2019: Stepping it Up

Americans, facing a barrage of conflicting guidance (masks, anyone?) and an uncoordinated response, figured out what steps to take, did them, and demonstrated the kind of selflessness an objective person would expect.

Apr 2019: Karen

Despite no real need for heavy handed enforcement, and negative value from preventing distancing at the extreme, the social justice warriors, mostly, transformed into a an army of cop-calling, finger pointing, neighbor outing scolds.

The New Politics continued, with experts in heavily populated blue states accusing lightly populated red states of a reckless selfish ignorance that would kill everyone.

Some people blamed China.

Some people blamed Trump.

The HCQ nonsense.

The bleach nonsense.

The “my Country is better than your county and will be spared” nonsense.

May 2019: The Silent Awakening

Through a combination of some states partial reopening, the de facto reopening that’s happening as people discover that in the broader population, the disease isn’t nearly as deadly as we assumed, compliance with “distancing” began to drop. About 6 weeks into this, the second wave didn’t seem to hit.

Jun 2019: Anger

Then country, indeed the world, is angry. Angry about Covid19. Angry about the lockdowns, job losses. Angry about the unfairness of government picking economic winners and losers. Angry at the rich getting richer, somehow. Bad science. The lack of lockdown compliance, and ignorance that will kill everyone.

Anger, searching for a victim it will stick to. Any victim.

The notion that the entire world is suddenly united in at the injustice that is black life in America, and not just angry and dangerously erratic, is nonsense.

Dangerous nonsense.


Posted in Uncategorized | Leave a comment

Corvid19 In an Age of Bullshit; Hold the Bullshit.

Last week, the person credited with Sweden’s Covid19 public strategy, Anders Tegnell admitted “There is quite obviously a potential for improvement in what we have done.”

With the wisdom of hindsight, it’s unlikely that anyone would have not have put the entire vulnerable population in 5 star hotels with room service, quarantined them, educated people about what was known/unknown about self-managing risk, and otherwise moved on with life.

This would have saved many lives and jobs, and would have cost vastly less than what we did.

Why we are not doing this now is less clear.

But little was known about Covid19 four months ago. We knew that it was highly infectious, lethal in ways not previously seen, and spread easily via droplets suspended in air. Had it spread and killed in the broader population in the same ways it does in a vulnerable population, the results would have been tragic on an Old Testament scale. Early worst case estimates of millions dead might have been low.

And so an expensive, all-hands on deck response was entirely reasonable.

We now know that danger to any one individual varies dramatically, and that a sorting into high/low risk groups is easy. The largest group, under 60 and in generally good health, are barely vulnerable at all, and that folks older than 60 with diabetes and/or heart disease are horrifyingly high-risk.

One unanswered question remains how effective are various mitigations?

Some welcome the failure of Swedish response as proof of something. It is not clear what we would have learned had Sweden had the same outcomes as Norway and Finland. Or how the puzzle is simpler now.

It’s difficult to know exactly what the state of the art in science-informed public policy re: Covid19, but it seems to be that the virus will naturally spread unchecked though the entire population linearly with time, that our policy has prevented this disaster so far, that it is the only thing preventing it from happening again, and will continue to happen until a vaccine or herd immunity saves us.

Given that we are only at 10-20% of people with antibodies, state of the art policy promises us heavy lockdowns for 3 decades, or finely tuned degrees of lockdowns, constantly changing, for 7 years.

If you somehow burned out wave 1, just one infected person will restart everything.

Why anyone would welcome such a scenario is unclear.

Fortunately, there are lots of reasons to doubt it. The much modelled wave of new death from some states partial reopening, 3-6 weeks, ago has yet to materialize.

All of the rules went out the window if you were with Antifa.

For reasons we don’t understand, at least in the first wave, the infection rates seems to top out at about 20% of the healthy, younger population.

At some level, even we didn’t believe it. About 60% of the economy continued to function. The government picked winners and losers, as we asked them to, but overall many things continued to operate. Why the virus cares if you are bowling or shopping was never made clear.

This thing was never serious enough to shut down dope stores.

Posted in Uncategorized | Leave a comment

Absolute Truth is Good

I was raised as a Protestant Baptist. Among mostly kind, well-intentioned, community minded folks. The churches ran outdoor camps, canoe trips, youth groups, exchange programs and other activities that helped many people, including me, explore the world surrounded by decent people.

It wasn’t all good. I found myself trapped at youth camp with leaders overzealous to further convert people. Some of my early church teachers seemed genuinely frightened by life and death, and shared that fear with young people in a, perhaps, not helpful way. And church was boring. And long. Really boring, and really long.

But nothing really bad,  and lots of good,  happened.

My first break from this belief system occurred when I learned that Santa Claus wasn’t real, around the late age of 8 or so. The frightening thing about this wasn’t the realization that a jolly feller didn’t come down chimneys with gifts, because obviously this wasn’t true. The frightening thing was my sudden realization that I had been willing to suspend belief in what I observed in the real world to believe in an imaginary world. Indeed, this event was the first of two that damaged my belief in mystical things, something that I never really recovered from.

The second event, not counting the release of Andrew Lloyd Webber’s Jesus Christ Superstar, was psychedelic drugs, in my teenage years. What I discovered was that there was an easy path to deeply mystical feelings that were more intense and real  on a whole new scale. And for a few years I was convinced there was some real truth behind this, something that I now believe is dangerous nonsense. Although, interesting. These days, I think anyone who chooses never to take psychedelic drugs is blessed. And many people who choose differently, differently blessed. One caution: you can’t go back. Psychedelics are in many ways far safer than the witches brew of other poisons people take to mess with their minds. But they alter your thinking. Forever.

I have no doubt that psychedelics made me a better engineer and problem solver. But also, maybe a bit nuts.

Post teenage years, I embraced science and engineering and reason as the path to truth, and rejected Christianity. And science is fun like drugs and far more amazing. It is, and remains, the shit.

About a decade later, I started to think about history and the development of societies. And much to my surprise, I noticed that many of the core values that shaped societies that successfully elevated humanity from a state of brutish nature (for a while), were rooted in Christian belief. Christianity wasn’t the only religion to do this, but it was an important one.

At that point I made some peace with Christianity, and decided that even if the specifics were unbelievable, and, perhaps, there was no God, much practical good flowed from it. Maybe. But I still held on to what I now understand was just bigotry, that Christians were, if not actually feeble-minded, willing to delude themselves. Sometimes dangerously so.

I took actually working closely with brilliant engineers who also happened to be Christians to make me question this unpleasantness. I encountered deeply thoughtful,  logical, and decent people who were not in any way afraid to share their beliefs and their the thought process behind them. They knew exactly when they had become comfortable with the suspension of disbelief, and why. Basically, it was a belief that absolute truth was unknowable, and that the choice was nihilism, or a belief system. And that thier choice of belief system was net positive for them. They offered considerable evidence of truth, but ultimately made no argument of certainty.

I was unpersuaded. But considerably less judgmental.

In my late 30s and 40s, I had learned enough about physics to conclude that what we cannot see, or directly measure, or really know from our observation position on planet earth was a road to insanity. At least it would be for me. And this observation didn’t in any way damped my deep awe for people who could understand quantum mechanics, relativity, and multiple dimensions.

But I was unpersuaded. Indeed, I reached a point where I decided that science and engineering, while working magic on a grand scale to elevate humanity, wasn’t ultimately going to answer the ultimate question of life, the universe, and everything.

Some people seem to argue with passion that the earth was created in 6 days, by a benign, eternal, all-knowing omnipotent creator. Who got tired, and need rest on the 7th day. And all of that led to jet airplanes, atomic bombs, iPhones and the Pet Shop Boys.

Somehow, this just seems too bizarrely improbable (to me) as to be likely.

Other people seem to argue that from nothing, a completely random explosion happened, that coded into the very structure of everything mechanisms that would eventually lead to airplanes, atomic bombs, iPhones, and the Pet Shop Boys.

Somehow, this just seems to bizarrely improbable to as to be likely.

Others argue that airplanes etc. are the result of a gazillion raised to the power of a bazillion random accidents, the vast majority of which led to nothing.

‘Nuf said about that.

Throughout the journey, I’ve never lost a deep fascination with nature boarder lining on the mystical. It’s gift, a blessing that I treasure.

I’ve long thought it was also a clue.

To what?

Oh hell.


Posted in Uncategorized | Leave a comment

Helpful Flying Information re: Expedia, Fiji Airways and AIG Travel “Insurance”

At least one of you, maybe two (certainly not AIG – yet) have emailed me one of those surveys marketing sends out that you aggregate into scorecards and use to decide budgets, bonuses and promotions etc. I apologize but have not responded yet.

Consider this to be my response. Consider it also to be my non-response to your email.

Full Disclosure: The critical business of the airline industry is to move people and things to far away places of their choice, very quickly and very safely. Given the physics, economics and regulation involved, I remain in awe at how well you do this. And it just seems to get better YoY.

Well done. And thanks. Nothing I write below should detract from this accomplishment.

What I often have trouble understanding is why, given my love of travel, and arriving at places of my choice quickly and safely, I often leave the airport angry.

This time, it’s hard to know who to be angry at, exactly. You seem to be working together.

But I do know why.

A bit of background: I’m in year one of retirement. Over the course of a business career, I flew several business flights and several family flights each year. While I’m no 1,000,000 mile salesperson, I am not an especially sophisticated bargain hunter (as we shall see) and often flew business on the corporate dime. I collect miles but have never redeemed them. I’m guessing I’ve been a decent customer, profit-wise.

Over several hundreds of flights over decades, I’ve cancelled or changed perhaps five. Certainly fewer than airlines have changed against my will often causing me to miss my connection or selected and paid for seats. Hold this thought please.

I have a bad back. Recently, I booked a trip to Fiji for two people and I choose business class, despite the fare being 5X the price. But I knew the deal. Sit in agony and humiliation for 10 hours or pay up. So I did. And I naively assumed that anyone paying the price of a decent used car for 2 airline seats might be treated with fairness and some graciousness. But taking no chances, I bought Travel Guard ™ Trip Protection Insurance from AIG, through Expedia, for the price of about half a coach ticket. Each.

As things turned out, it became important for personal reasons to return one day sooner than we had booked. I made this change about 10 days before we left.

I called Expedia, and they cheerfully and empathetically explained that they could help change the flights, but that typically airlines would charge between $100 and $300 per ticket for the change. I had noticed that my flights had gone down in price by about $800 / ticket, so I hoped maybe the reduction might cover this.

I bet you can see where this is going.

It did not cover this.

Expedia customer service transferred me to Fiji Airlines, where they explained that the change fee would be nearly as much as the cost of a new ticket at the new lower rates, or 10X to 30X the fee Expedia described. I sensibly declined this madness but was told by both Expedia and Fiji that I had cancellation insurance and could file a claim.

Technically, this was true.

I filed a claim.

Expedia’s marketing department sent me an email telling me that the price of this trip had dropped even further. I was confused as to how this was helpful.

The claim was denied.

I’ll not list AIG’s response except to quote this: Please refer to the policy wording, which states in pertinent part:, followed by a list of exact reasons why a claim may be paid. My reason wasn’t on the list.

In retrospect, it seems obvious given the effort airlines go to advertise low prices, while charging higher prices, and to charge higher prices for exactly the same thing to different people based on their sophistication, ability and willingness to pay and good or bad fortune, that the insurance industry couldn’t be the backstop for this. People would game the airlines, who are gaming them, using insurance by doing things like cancelling higher priced tickets for today’s prices.

We couldn’t have that.

I guess.

In retrospect, I feel a bit naïve for even assuming my claim would be filled. In my defense, this was confusing:

TRIP CANCELLATION 100% Trip Cost Per Insured

TRIP INTERRUPTION 100% Trip Cost Per Insured

The key here, if you are paying close attention, is that something outside of your control has to cancel or interrupt the trip. This is helpfully more clear in the documentation of the refused claim than the documentation during the sale.

I’m sure Expedia didn’t know this when selling me the “insurance”, nor did they or Fiji Airlines when referring me to AIG during the change process. Or pricing the change.

Anyway, it all might work out. Here’s where things stand now:

From: TG STP Claims Doc 1 <>
Date: Tuesday, March 10, 2020 at 11:33 AM
To: Mike Zintel
Subject: Automatic reply: AIG Travel Travel Insurance Case XXXXXXXXXXX

Please do not respond to this email. Thank you for your correspondence. An analyst will review your documentation and contact you if further information is required to process your claim.

Ok, I won’t respond to the email.


Posted in Uncategorized | Leave a comment

60 Years of Social Engineering (was: The Cycles of Acceptable Liberal Thought)

American Liberalism began with the generations immediately following the one that fought and died in WWII. The central belief was that society had progressed both in material comfort and enlightened thought to a degree that life should, and could, be kinder, easier & freer. Indeed, it was the many of the old values holding us back from these things. American Liberalism in the 70s was optimistic, hopeful, curious and fundamentally decent.

First Wave Feminism was focused on equality of opportunity. There was no assumption that men and women were interchangeable, but grossly unequal in political and economic power, including even the right to vote.

First Wave Feminism was successful in correcting the injustices it targeted. Celebrate this.

Vietnam brought the choice between the old and the new into sharp relief. You could be for being drafted to slog through the jungle, schlepping 60 lbs of gear, nursing a persistent and painful foot fungus, killing and being killed in a senseless, amoral war, or you could be for dropping acid, smoking weed, listening to the Mamas and the Papas, Neil Young, and Buffalo Springfield while having casual consensual sex.

The correct choice here was not only clear, but also an excellent deal for most men.  They became lefties for life. And why not.

Second Wave Feminism taught us that women and men were biologically identical, with one observable exception: only women could bear children. Any behavioral, aptitude, or preference differences between adult men and women were entirely the result of socialization, which was bad because it was limiting for women. Nobody really believed this, but feminism and liberalism were achieving good things, and so folks started on the long road of pretending to believe.

Second and Third Wave Feminism were primarily concerned with violence against women, sexual freedom, and employment opportunity and income parity with men. These movements wanted women out of the homemaker, caregiver, wife and mother roles, if they so choose. Celebrate this.

Between the 60s and AIDs, sexual liberty exploded, but generally against an assumed to be forever value system of committed, loving, loyal relationships being desirable and important.

This value system remains true for the middle classes and above. The poorer folks tend to actually do what the elites claim to be good for them. The elites for the most part talk a lot smack but practice traditional values. So far.

Women, having attained the holy grail of being able to be shipped off to war or work all day in a factory or office, discovered what men had long known. It isn’t that fun. Men don’t work 50 hours a week for an unpleasant boss for their entire lives without complaining because it’s spiritually fulfilling. They do it because it is expected, and supports their families. And their hearts break as they watch their children grow up from a distance. Generally, they suffer in stoic silence. And retreat into sports and beer.

One of successes of liberalism was civil rights. Celebrate this.

Along the journey, we were taught to know that gays were born this way. Most folks didn’t really know or care if this was actually true or not, but it was a helpful way to break through the stigma that homosexuality was unnatural or actually a sin. What was never explained was how to square the fact that men and women are certainly not born this way but entirely shaped by their upbringing and life experience, but gays were.

But  no matter. When you are inventing an entirely new religion you have to make some compromises along the way for the greater good. Like every other religion.

Somewhere along the way, every vice became a disease. Nobody was responsible for anything anymore. We’re all victims of something or another. Hold that thought.

At this point, I would like to write something, likely critical, about Critical Race Theory, but I can’t understand it. Something about epistemology and white privilege and unending structural racism and oppression. As huge progress against actual racism and oppression were made and broadly welcomed, we are taught that it is simply impossible not be an opportunist, oppressive racist, if you are white, or to not be a victim of this if you are black.

Don’t celebrate this.

STDs, children, jobs, aging parents, and the greedy 80s put the brakes on American liberalism. Indeed it would never return as it was understood at the time.

With the dawn of Forty-Seventh Wave Feminism, things have a gotten a little more complicated. Over time, denigrating fathers, Christians and family itself, and systemically feminizing boys, and using victimization as weapon have replaced the hopeful, optimistic liberalism of the 60s and 70s.

Many young men, having been feminized from birth, lacking in positive masculine role models and organizations, try and fail to make sense of drunk hookup culture, Tinder and Grindr, and “lol yusss. I’ll be over w da females soon” Mattress Girl and decide, rationally, that internet porn and video games is a safest option.

Many 30-something women wonder why there aren’t any good men left, and increasingly expect other taxpayers to support them as there is no husband to do so.

Intersectionality is liberalism’s (now renamed progressivism), New Testament-esque grand unifying theory. Intersectionality teaches us that America consists of 3 demographic groups, and that the values and actions of two of those demographics are constant, determined solely by skin color, gender and ethnicity.

The first demographic is white men and Christians. This demographic has no redeeming qualities, but this is not entirely their fault (or fixable), because they are in fact born this way. It is socially acceptable to speak of these folks in denigrating and unkind terms.

The second demographic is Jews, Asians, and immigrants from India. This demographic shares many qualities with the first, but it is generally not socially acceptable to speak of these folks in denigrating and unkind terms. Celebrate this. A bit.

The last demographic is everyone else, with the shared quality that they are born this way victims of the previous two groups.

Intersectionality is an entirely rational theory. Being included in government, industry and academic official lists of approved victims, affords one with considerable structural favoritism. And being excluded, discrimination. And while this might seem unfair , indeed just old fashioned prejudice to some people, Intersectionality teaches us that it is not. Because oppressors and victims are born this way, and will remain this way until death. Nothing can ever be fixed. And yet we must continue to try.

But with so much money and power at stake, and obvious in and out groups among the victim demographic, things can get a little tense.

Unlike in the past, when men and women were born interchangeable empty vessels for society to program, now men and women don’t actually exist. There are no genders, or infinite genders, we are taught.

Unless the lack of traditional women-ness interferes with any legal or structural advantage women have earned. Like say, child support, women-only places or Title IX. Then it’s just nonsense.


The pronoun nonsense will be short lived. Nobody believes the gender nonsense.

Gender is a mildly clinical synonym for sex, when used as a noun.




Posted in Uncategorized | Leave a comment

D850 Notes For Underwater Videographers and Fireworks Photographers and Others

The topmost settings that control focus are AF-mode and AF-Area Mode:

  • Note that both settings have different states, and different algos between Live View and Still
  • Note that the algos may also be different between Live View Still and Live View Movie (I suspect this)
  • Despite different states and algos, the same camera controls change them; some of the wee icons are different; clues

To defang AF (i.e. turn the worse than useless advanced nonsense off) and make it actually work in Still / Not Live View:

  • VR – your choice
  • AF-S, Single Point Mode; lock the point in the center
  • Shooting: Aim the focus point, half shutter press and hold, reframe and fire
  • AF-ON seems useful as it does the same thing as half shutter press, but isn’t because the camera will refocus on shutter press; useless
    • Don’t forget this as you keep reading
  • In the viewfinder, “Single Point Mode” is “S” when AF-Mode center button is pressed, but icon on Info display (not Live View) is different; tricky
    • Too bad you can’t see wee viewfinder icons underwater; try to get this right topside
    • On Info display it’s the smallest rectangle-like thing

Underwater Still / Not Live View Metering

  • Aperture Priority (why does any other mode other than M actually exist?)
  • Matrix (the icon with the dot and the four weird shapes around it
  • Auto-ISO, usually
  • Camera tends to blow out highlights; correct with +/-; watch the histograms; don’t forget +/- settings on subsequent shots
    • Highlight-Weighted metering works 80% under water – perfectly – and underexposes by 5 stops 20% of the time
    • Last time it failed for me was nurse sharks feeding; do you feel lucky today?

Still / Live View

  • Fear leads to anger. Anger leads to hate. Hate leads to suffering. Don’t use it

To defang AF Video  / Live View:

  • VR off; make sure you do this topside; this is critical underwater
  • AF-C, Normal Area AF, which seems to be close to Single Point Mode in Still
    • Note you must be in Live View / Video to set; otherwise you are changing something else; surprise!
    • Icons (and algos?) are different from Still mode;  small rectangle + NORM is what you want
  • Shooting: Aim the focus point, lock and hold with AF-ON, reframe and fire
    • Note that if you have remapped Shutter to Video Record with setting G1, the camera will refocus on shutter press; bad
      • One solution: don’t map it; but it’s handy
      • A better solution, setting A8 AF-activation to Off
      • Now you must use AF-ON, even in Still; don’t forget this!
      • If you never shoot video, this will just cause you to forget to focus on Still
  • Note that in Live View, cycling through Info gives lots of useful Info, including AF-Area Mode and histogram
  • AF-F, which stands AF-Fucked, doesn’t do what the manual says it does
    • It never, ever, stops focusing; not on half shutter press, not on record mode; never
    • This is excellent if you want video of the camera focus-hunting, non-stop
    • Is this a debug mode that got left in by accident?
    • How would this ever be useful?
  • There are a lot of complicated options for limiting AF functionality in settings, which makes me believe that Nikon knows it doesn’t work
    • A9, limit AF-Area Mode, seems handy
    • But note that it works in Still / Not Live View Mode only; doesn’t work in Live View; more trickery

Underwater Video  / Live View Metering

  • Nothing unusual from Still mode except that everything must be setup in Live View
  • Aperture Priority
  • Matrix (the icon with the dot and the four weird shapes around it)
  • Auto-ISO, usually
  • Camera tends to blow out highlights; correct with +/-; watch the histograms; don’t forget +/- settings on subsequent shots
  • Highlight-Weighted metering, do you feel lucky today?

Fireworks Stills On a Tripod

  • Aperture Priority, Matrix
  • ISO 400-800
  • f8-f11
  • +/- minus one or two stops
  • This will generally yield a multi-second exposure that gets background + trails
  • Manual focus; tape at infinity

Fireworks Video On a Tripod

  • Aperture Priority, Matrix
  • Auto-ISO
  • f8-f11
  • +/- minus one stops; less correction is more
  • Manual focus; tape at infinity


Posted in Uncategorized | Leave a comment

How Hard Indeed

Amazon recently completed a genuinely massive migration project from Oracle to AWS databases. This is an impressive accomplishment, demonstrating both technical prowess, and sustained corporate willpower. The data model of a large system permeates the entire system, and over time becomes a source a great fragility. The broader industry refers to such projects as Digital Transformations. Outside of the technology industry, these tend to be changes to existing systems to accommodate new business models, often  models that are enabled by B2B and B2c experiences on the internet. Connectivity touches everything these days: supply chain, shipping logistics, support and service,  manufacturing, R&D, and of course, disruptive new models enabled by connected mobile apps.

The last project that I led was a digital transformation. Multiple existing, overlapping  systems, many operating at scale, were causing a highly fragmented customer experience that negatively impacted both sales and support, but also the internal view of the customer.

At least at the beginning of the project, there seemed to be broad agreement that converging and simplifying these systems would lead to real customer satisfaction and increase sales.

My experience as a technology leader up to that point had been working on, or designing internal systems level platforms or products offered for sale. I had a good track record, including at least one system of world class scale, availability and correctness.

How hard could converging a bunch of user space apps be?

Stunningly hard, as it turns out. I gained a lot of understanding and respect for what IT folks live day to day.

If you find yourself in this situation, consider:

Double and Triple Check Your Assumptions About Business Value

I think we got this one mostly right. To do it required a search across the company for the handful of folks with the history and pulse on how things really work and should work Studying the existing systems, and listening carefully to the folks who understand them, helps, at least to understand how things work now. Despite working at the same company for 25 years, and being knowledgeable about the internals of many  of our products, I discovered systems that had trained over the years to subtle and devilishly complex business models. This should have been a red flag: Can we change all of these models? All at once? In what order? What will be required, training-wise? What if the changes negatively impact the business? Will they?

Get the Reference Data Model Right

Once you have reverse engineered the existing systems, it’s time to find the data. Sometimes it will be densely packed in a cluster of SQL databases. This is a gift, although it might not always seem so. Sometimes the data will have been copied (“denormalized”), and nobody is quite sure who is master now or how things stay in sync. Or if they do. If your system is old enough, there will be multiple brittle, schema transforming connectors between islands of data, and attempts in time to re-convergence in form of layers, which often contain copies of data.

The reference model is an exercise to identify the set of cohesive (“like”) and decoupled objects which map to real world business and customer things, virtual or physical, from the viewpoint of how would we build this now, with perfect knowledge of the requirements? If the data model is right, the object models and layering will often fall out naturally, and the system will evolve easily over time. If it is not right, each wave of smaller and smaller changes will become increasingly more expensive, and large changes simply impossible. The system will also not be auditable, or provably correct. Or actually correct.

This is the first step in figuring out where, ideally, you would like to go. A side effect of this exercise will be a much deeper understanding of the existing system and the business requirements.

Inventory Everything, Classify It, and Make a Plan

It’s tempting to green light a complete system reboot. Rewrite it all. Engineers love writing new things, with new tools. Iterating on crusty old things, not so much. It’s a mistake. There will be parts of the system so complex and brittle that the correct strategy is to stabilize them, wrap them and put them in surgical maintenance, with a plan that they live forever. This will do some damage to the reference data model. Plan for it. Other parts of the system will be in maintenance until they are replaced entirely. And some parts can and should be written on top of new objects and APIs. The structure and sequencing of this is critical, and will evolve over time.

One thing to pay particular attention to is scale bombs. These are systems that have a component that scales non-linearly with offered load, in a business environment where offered load is growing. Sometimes, non-linearly. One strategy can be to scale up with hardware and hacks, until these systems can be replaced. Sometimes you can get just the nasty load off the old systems, and let them run forever with smaller loads. Sometimes due to schema issues, the scale problems are not alleviated until the all of the load is off the systems. Complete shutdown. Modelling and calculating and monitoring time to explode is critical. These system often take months or quarters , and the very best engineers to fix. And if the business stops, well, the business stops. It won’t be good.

It is also important to get credit for this work. Otherwise this cost will get booked by default to the transformation project, which will, on its own, likely go over budget.

Layers and Re-platforming

There is nothing wrong with Service Oriented Architecture. It is a prerequisite for a decoupled distributed system. What you have to be a little careful of is a layer that adds SOA to part of an existing system where the data model really needs to be changed, but isn’t. Engineers like building layers. The new top edge is often more cleaner, easier to understand, and the systems above it, more maintainable. Once they are rewritten.

In cases where you have a system in surgical maintenance, and the data model doesn’t need to change, or can’t be changed, SOA API layers can make sense.

There are two forms of SOA layers. The simple one re-exposes a chuck of the existing data model, semantically unchanged. The danger here is that you burn considerable time, money and political capital on a disruptive change that accomplishes nothing toward the real business goals of digital transformation. The more complex one attempts to re-converge disjoint systems in the API layer.  This can work, and is easier than fiddling with the base layers of the system. But there are real ceilings to this approach, and getting the balance right between fundamentally fixing the data model, and faking it, are some of the most important architectural decisions you will make.

A similar danger lies in re-platforming. Engineers like to re-platform. Move to modern tools. Virtualize. NoSQL. node.js. Re-platforming, in the absence of fundamentally evolving the system toward real business goals, is often a distraction.

Charter Creep

On this project, our team acquired a reputation for being good at managing crusty old systems. Unsurprisingly, in retrospect, there were people at the door willing to hand us crusty old systems, with headcount and budget. We took them. Many team members were happy, because they had bigger teams and bigger charters. Many of these systems were taking real load, with real customers, and were quite important to the business. But as it turned out, many of them were time-bombed, or audit-bombed, or compliance-bombed, and contained all manner of other surprises. Take on too many of these, and you will lose focus and turn your digital transformation project into a IT shop. But without the discipline that real IT shops have.

The cost of these things will get booked by default to the transformation project, which will, on its own, have gone over budget.

As teams get very large, in the best case you get more confusion, and in the worst, duplication and deliberate sabotage.

Distributed Apps Need an Architecture

I should have gotten this right earlier, but didn’t. I was focused on the evolving data model, service componentization, layering, making sure we were tracking to convergence over time, and that real business problems were being fundamentally enabled. What I missed is that distributed apps, like distributed systems, have errors. In the case of apps, often subtle and complex errors. Sufficiently complex systems cannot in practice be debugged completely. And if they could, they wouldn’t stay that way for long. It is critical that an architecture for error handling, retries, orchestration, compensation, and that the app – left to run on its own – converges toward correctness. A sort of OK middle state is that errors are caught, and corrected manually. A bad state is that the app silently converges toward incorrectness.

I now believe that most complex distributed apps silently converge toward incorrectness. And Support, and Audit catch and repair some it. You can experience this up close if you try a data layer migration of a layer that’s been around for a good while. A good chunk of it moves, but then the long tail of schema errors festers. It’s often surprising the app seems as correct and available as it does, given the crap in the databases.

It can feel like an unwinnable battle. But not being intentional only makes it worse.

Organization and Teams

For much of my career in the capacity of technical leader, I got lucky. I worked at company that was attracting top talent, and I was working on things that made creating opportunity to learn and grow relatively easy. Over the years, many talented people saved me from myself. This project was different. While we had many experienced and exceptional engineers, we also had a large percentage of early in career people. It was a little hard to recruit and retain the very top tier of experienced talent. The app, while critical to the business, was correctly not the primary focus of the CEO, and wasn’t aligned with our core products and services in the way that, for example, fulfillment or search is at Amazon. Or the feed and ads at facebook, mobile and maps, at Apple and Google. We had scale challenges, but for the most part mega scale was not required. If you were primarily interested in learning new marketable skills, we were not at the cutting edge.

Scale I got.

In a climate like this, you got to deploy your best people very thoughtfully, invest in growing people and remember to explain at team scale what we are doing, and why it matters. Again. The latter was not my comfort zone, and the organization suffered for it.

There exists another organizational challenge in large teams on such projects. If you create a maintenance team, and a new thing team, then the new thing folks miss out on critical knowledge about how things actually work. And if you combine them, you can lose focus on the system in maintenance. Or focus too much on it. We (mostly) erred on the side of combining them, which I think was the best choice. At least at the beginning. Toward the end the folks working on the system in maintenance were beginning to working on migration tools.

The investment and opportunity cost to running the business on the system in maintenance can be very high.

The cost of these things will get booked by default to the transformation project, which will, at this point, be way over budget and late.

At some point, you have to rally to a milestone, suspend disbelief, finish, migrate, plan for and deploy it. Successfully. Without toasting the team. Much. Then, do it again. Until you are done.

Rolling it Out

There was an internal debate about the degree of business process and policy change that the new system (only) would enable. This had the benefit of getting people excited about the transition. But in practice trying to roll out a new system, run migrations, train people new systems and tools, and implement policy changes was too complex, and too risky. I pivoted to a strategy of building near perfect backward compatibility into the new system, rolling it silently, and then demonstrating how new business process and policy (often ready to go under the covers) could be implemented quickly. This was effective, but added a lot of cost.

All up, the cost of keeping the systems in maintenance running, migrations and backward compatibility consumed the bulk of the teams capacity. More than half. That, I hadn’t anticipated. IT folks are laughing.

Executive Support

For the first half of the project, the management chain above me was short, and reported to the CEO. My immediate manager, and his manager understood the strategy, the value and the complexity, and had confidence in the team. I took this for granted, and focused on building things. Until it was gone.

If you have read this far, perhaps you wondering if this was ultimately a success. I retired 5 months ago, after being a technical leader for 6 years on that project. And we were not done. So no – I didn’t feel successful. I would have done things very differently, with the wisdom of hindsight.

That said, we changed business strategy is a very big way, mid project. It was a necessary and correct decision, but cost us 2 years of investment.

We defused a number of scale bombs and improved correctness and availability, and kept a large and growing business growing.

The systems in maintenance teams delivered flawlessly. They kept things running, at scale, and waited for team new thing to catch up. They were always helpful.

The new system supported a fast growing new business with immense scale and unique business model requirements. Successfully.

Existing systems were starting to be migrated to the new system with success.


And a very good team is still working it.


Posted in Uncategorized | Leave a comment

The Watchful Internet

Do you ever wonder how it is that you are browsing, and a few seconds later, as you browse Amazon for McGregor socks, and Amazon shows you an ad for a Caterpillar 797 Heavy Hauler?

Who doesn’t want a Cat 797? But this is a little creepy.

Sometimes the industry attempts bamboozle you by explaining that such events are just the bizarre little coincidences, enabled by the magic of machine learning (ML) or artificial intelligence. The claim is that they are not tracking you, they are tracking a group of people who exhibit similar actions and interests, and that ML can predict your next action / interest from the actions / interests of the whole group. In the simple case, if the group is searching for morning sickness, sudden infant death syndrome , then the ML will predict that you will soon be searching for diapers, car seats and baby clothes.

While ML can and does work this way, and is used to make surprisingly good suggestions for video games, movies, music, and in the case of Amazon, things you want but didn’t know you wanted, this isn’t how the 797 scenario works.

They are tracking you. But who are they?

Let’s say we want to set up an ad serving business. We’d need to build an ingestion pipeline, so ads could be submitted for sale. The UX / APIs would likely include desired demographic, and prices willing to be paid for better and better demographic targeting. In the most sophisticated systems, these things would be auctioned in realtime to the highest bidder.

Once we have an ad inventory for sale, we’d have to find to places and people willing to serve the ad. The easiest way to accomplish this is to pay people for serving ads. In web pages, and apps – usually mobile apps – since neither Windows nor MacOS rely (much) on ads for monetization. We’d have to build a billing system that registered views (“impressions”) and clicks.

Then we’d need to build a visual ad serving component, one for each of the browsers and important mobile app platforms. All this thing has to do is reliably display an ad, and record if it was in fact displayed, if customer clicked on it, or how far they watched in the case of video, in order to bill the ad supplier the correct amount.

At this point we’d be in business, but not likely profitable. Our ads wouldn’t fetch a big price, because we are not telling the ad supplier much about the target customer, nor were ads targeted at likely interested people.

What we need next is the ability to learn about individual customers.

That’s where trackers come in. A tracker is a bit code that lives in the browser, or a mobile app, continuously watches what you do and uploads it to the tracker / ad network for storage. There are only two hard problems in building a tracker network. The first is reliably identifying each unique customer, and the second incenting web page owners to install your tracker. The latter problem is easy, just share the revenue with them.

The identity is a little bit hard. Unless you are visiting a site where a login is required, the web server only gets an IP address and bit of information about your operating environment (browser+version, operating system+version, cpu, etc..) Seemingly small bananas, privacy wise.

Without going into the details here, most trackers can derive enough information about you and your computer to uniquely fingerprint you. This means that they can assign a stable identifier to you and computer, such that subsequent browsing / app usage on the same computer can be correctly attributed to you. They assign you a identifier; let’s call it a somewhat_anonymous_id. Mine might be 4656879900786 (on one computer).

Now let’s say we’ve got our tracker and our ad serving component running in lots of pages. As I travel around the internet, the tracker follows me and uploads lots of personal information, and stores it via my somewhat_anonymous_id, 4656879900786. When I browse to a page, or app, that has our ad serving component installed, the ad component generates the exact same somewhat_anonymous_id, 4656879900786, that the trackers have been generating. It asks the service to find the ad with the best match / highest impression price. The ad is served, and telemetry monitors click through, video watch time, etc. to derive the final price.

In theory, you see the creepy Caterpillar ad and your privacy is protected. The seller of the ad, the ad serving system, and the page owner who served the ad all don’t know who 4656879900786 is.

Or do they?

The next problem to be solved is to create  a stable identifier that can span devices. The easiest, but not the only way to do this, is to do a cloud JOIN across multiple telemetry streams, correlating information likely to identify a unique person across devices.

Imagine that you have search records for rental cars, hotels and flights to the Balearic Island of Ibiza, with times. And you have, via a somewhat_anonymous_id, credit card history (yes, banks sell this). And maybe some mapping searches, which of course include location information. It’s not too hard to look across all of these sources, all with different, somewhat_annonomous_ids, and conclude that this is the same person. They can then be assigned a master_anonymous_id, which can be associated with all the somewhat_anonymous_ids. This process can be done in batch jobs, and tuned over time as new information arrives, but once the association is established, it can be used in realtime.

The final step in this process is to JOIN the master_anonymous_id with an actual human by name. The easiest way to do this is to incent customers to provide it with value. Stored passwords, form-fill, credit cards, checkout by …, login with facebook …, are valuable things, often requiring a login. Email, Twitter, Instagram, cloud storage all require a login. In theory, this is all customer-friendly and private.

But it gets a bit janky.

Once a password is stored in the browser, it is available for use by code issued from the same source that caches it in the first place. That code can run without your consent, and without actually even generating user experience. There are lots of valid reasons for this, but it also opens a door for trusted code to behave as a tracker with full credentials.

None of the above assumes anything particularly nefarious (other than the invisible cloud JOINs), and has valid business and customer reason to exist.

The facebook graph API was believed to upload information on pages where it was enabled, by not even used. I have no way to prove this was happening, but it is technically possible.

Google has direct access to all of your searches. They have a little more visibility into your behavior via ad “referrals”. Facebook has all of the content you knowingly uploaded. What is exceptionally valuable to both of these companies (and Amazon), is knowledge of everything else you do, across the web and within applications. Something they don’t get without intentional effort.

Long ago, Google released a free analytic suite called Google Analytics. It’s very good. Of course to work, it has to upload detailed click stream data. The value to Google is visibility into the rest of people’s browsing patterns post search, at least at the somewhat_anonymous_id level. Google really wants you to login, and store your credentials. So badly, they will offer substantial amounts of free services in exchange for this.

Looking at Ghostery, I see that loads 15 trackers Disney, 10, National Review, 25, Slate, 30.

There seem to be more trackers than ad networks, so perhaps there is a business simply selling profile information to ad companies.

Fortunately, there’s government to rescue us from nefarious business. About 3 years ago the EU passed a law that required explicit user consent to use browser cookies.

That resulted in billions of clicks on things like this:

And it accomplished nothing. Trackers have not used cookies to fingerprint for years now. Clearing your cookies improves privacy only from the least sophisticated players. The security models of cookies makes the Caterpillar scenario difficult or impossible.

The EU’s GDPR law has a bit more teeth against facebook, Google and Amazon, but still, not much. I spent 6 minutes on facebook trying to find out how to delete all of information, as is required by GDPR. I didn’t find it. Perhaps they have argued that it is all necessary for the experience, and is exempt. Deleting it would paralyze my facebook experience. Losing my order history on Amazon would be inconvenient. And all the tracker and ad network information is basically invisible, anyway. How would one even begin to erase all the tracker data.

Hypothetically, if governments passed a law effectively blocking trackers, all of the advantage would shift to facebook, Google, and to a lessor extent Amazon. They are big enough, and have enough reach, that they are their own tracker network.


Everything in this post is knowable from published ML/AI research and other public sources.

Posted in Uncategorized | 5 Comments

How It Works

Liberals still don’t seem to understand how Donald Trump got elected, and perhaps will be elected again.

Ironic, because that’s how we got here.

Liberals believe that in the election of Barack Obama, they (ignoring all the independent and conservative vote for Obama) elected a gentle, kind, wise, intelligent, and very funny man to be president. And rednecks were so incensed at the idea of a black president that they elected Donald Trump as payback.

There’s a little truth here, but not much.

This how the game was played during the Obama administration: The Obamas took the high road, and said mostly benign, secular-pious sounding things that meant whatever you wanted to read into them. And then Holder, Van Jones, Rahm, etc. etc. went bare knuckles. And the entrenched liberal leviathan kept at it, under the radar.

It’s an effective  system. It works. Especially when the president is a brilliant, charismatic speaker.

And it should have continued working. Except that the we are the one’s we’ve been waiting for crowd could not control their arrogance. On the television, Keith Olbermann, John Stewart and Bill Maher, etc. etc. were basking in reality that that the really smart people were finally in charge. And they let everyone know it.

The liberals loved this. It was clever and funny. So clever that they though dumb rednecks didn’t even get it.

Conservatives found it a bit condescending, which it was.

Up until Romney / Palin. That was the point when the funny / clever cloak came off, and many powerful liberals exposed themselves as mean, vulgar, hypocritical narcissists. Somehow, they have erased this memory. Conveniently.

Liberals were angry about Congress. For the next 4 years, the country was subjected to non-stop barrage of condescending meanness. This memory is also gone now. Conveniently.

I never got the sense that Barak was angry. He certainly wasn’t the slightest bit angry about being black. He figured out long ago that the combination of black +  white guilt + genuine charm and talent was his ticket to the good life. $100M in two years after leaving office; what’s to be angry about.

But he was also in love with himself. Perhaps he didn’t plan to ridicule the middle of country, he just didn’t know anything at all about them. And as a life-long privileged elitist, how could he.

But he did.

And everyone followed.

Folks got angry. Anger was in fact the correct response.

And they elected Donald Trump.

Now everyone’s angry at everyone about everything all the time.


Posted in Uncategorized | Leave a comment